Merge remote-tracking branch 'upstream/master' into hardened-stdenv

6 files changed, 251 insertions, 121 deletions

diff --git a/doc/default.nix b/doc/default.nix
index 6a44587a31b..076703213e4 100644
--- a/doc/default.nix
+++ b/doc/default.nix
@@ -57,11 +57,11 @@ stdenv.mkDerivation {
       outputFile = "./languages-frameworks/haskell.xml";
     }
   + toDocbook {
-      inputFile = ./../pkgs/development/idris-modules/README.md;
+      inputFile = ../pkgs/development/idris-modules/README.md;
       outputFile = "languages-frameworks/idris.xml";
     }
   + toDocbook {
-      inputFile = ./../pkgs/development/r-modules/README.md;
+      inputFile = ../pkgs/development/r-modules/README.md;
       outputFile = "languages-frameworks/r.xml";
     }
   + ''
diff --git a/doc/functions.xml b/doc/functions.xml
index e6bb6b7deef..908e9571ed6 100644
--- a/doc/functions.xml
+++ b/doc/functions.xml
@@ -89,27 +89,27 @@ in ...</programlisting>
   <title>&lt;pkg&gt;.overrideDerivation</title>
 
   <warning>
-    <para>Do not use this function in Nixpkgs. Because it breaks
-    package abstraction and doesn’t provide error checking for
-    function arguments, it is only intended for ad-hoc customisation
-    (such as in <filename>~/.nixpkgs/config.nix</filename>).
-   </para>
-
-   <para>
-    Additionally, <varname>overrideDerivation</varname> forces an evaluation
-    of the Derivation which can be quite a performance penalty if there are many
-    overrides used.
+    <para>Do not use this function in Nixpkgs as it evaluates a Derivation
+    before modifying it, which breaks package abstraction and removes
+    error-checking of function arguments. In addition, this
+    evaluation-per-function application incurs a performance penalty,
+    which can become a problem if many overrides are used.
+    It is only intended for ad-hoc customisation, such as in
+    <filename>~/.nixpkgs/config.nix</filename>.
    </para>
   </warning>
 
   <para>
-    The function <varname>overrideDerivation</varname> is usually available for all the
-    derivations in the nixpkgs expression (<varname>pkgs</varname>).
+    The function <varname>overrideDerivation</varname> creates a new derivation
+    based on an existing one by overriding the original's attributes with
+    the attribute set produced by the specified function.
+    This function is available on all
+    derivations defined using the <varname>makeOverridable</varname> function.
+    Most standard derivation-producing functions, such as
+    <varname>stdenv.mkDerivation</varname>, are defined using this
+    function, which means most packages in the nixpkgs expression,
+    <varname>pkgs</varname>, have this function.
   </para> 
-  <para>
-    It is used to create a new derivation by overriding the attributes of
-    the original derivation according to the given function.
-  </para>
 
   <para>
     Example usage:
@@ -125,9 +125,9 @@ in ...</programlisting>
   </para>
 
   <para>
-    In the above example, the name, src and patches of the derivation
-    will be overridden, while all other attributes will be retained from the
-    original derivation.
+    In the above example, the <varname>name</varname>, <varname>src</varname>,
+    and <varname>patches</varname> of the derivation will be overridden, while
+    all other attributes will be retained from the original derivation.
   </para>
 
   <para>
@@ -135,6 +135,20 @@ in ...</programlisting>
     the original derivation.
   </para>
 
+  <note>
+    <para>
+      A package's attributes are evaluated *before* being modified by
+      the <varname>overrideDerivation</varname> function.
+      For example, the <varname>name</varname> attribute reference
+      in <varname>url = "mirror://gnu/hello/${name}.tar.gz";</varname>
+      is filled-in *before* the <varname>overrideDerivation</varname> function
+      modifies the attribute set. This means that overriding the
+      <varname>name</varname> attribute, in this example, *will not* change the
+      value of the <varname>url</varname> attribute. Instead, we need to override
+      both the <varname>name</varname> *and* <varname>url</varname> attributes.
+    </para>
+  </note>
+
 </section>
 
 <section xml:id="sec-lib-makeOverridable">
@@ -171,42 +185,18 @@ c = lib.makeOverridable f { a = 1; b = 2; }</programlisting>
 
 
 <section xml:id="sec-fhs-environments">
-  <title>buildFHSChrootEnv/buildFHSUserEnv</title>
+  <title>buildFHSUserEnv</title>
 
   <para>
-    <function>buildFHSChrootEnv</function> and
-    <function>buildFHSUserEnv</function> provide a way to build and run
-    FHS-compatible lightweight sandboxes. They get their own isolated root with
-    binded <filename>/nix/store</filename>, so their footprint in terms of disk
+    <function>buildFHSUserEnv</function> provides a way to build and run
+    FHS-compatible lightweight sandboxes. It creates an isolated root with
+    bound <filename>/nix/store</filename>, so its footprint in terms of disk
     space needed is quite small. This allows one to run software which is hard or
     unfeasible to patch for NixOS -- 3rd-party source trees with FHS assumptions,
     games distributed as tarballs, software with integrity checking and/or external
-    self-updated binaries.
-  </para>
-
-  <para>
-    <function>buildFHSChrootEnv</function> allows to create persistent
-    environments, which can be constructed, deconstructed and entered by
-    multiple users at once. A downside is that it requires
-    <literal>root</literal> access for both those who create and destroy and
-    those who enter it. It can be useful to create environments for daemons that
-    one can enter and observe.
-  </para>
-
-  <para>
-    <function>buildFHSUserEnv</function> uses Linux namespaces feature to create
+    self-updated binaries. It uses Linux namespaces feature to create
     temporary lightweight environments which are destroyed after all child
-    processes exit. It does not require root access, and can be useful to create
-    sandboxes and wrap applications.
-  </para>
-
-  <para>
-    Those functions both rely on <function>buildFHSEnv</function>, which creates
-    an actual directory structure given a list of necessary packages and extra
-    build commands.
-    <function>buildFHSChrootEnv</function> and <function>buildFHSUserEnv</function>
-    both accept those arguments which are passed to
-    <function>buildFHSEnv</function>:
+    processes exit, without root user rights requirement. Accepted arguments are:
   </para>
 
   <variablelist>
@@ -220,14 +210,16 @@ c = lib.makeOverridable f { a = 1; b = 2; }</programlisting>
     <term><literal>targetPkgs</literal></term>
 
     <listitem><para>Packages to be installed for the main host's architecture
-    (i.e. x86_64 on x86_64 installations).</para></listitem>
+    (i.e. x86_64 on x86_64 installations). Along with libraries binaries are also
+    installed.</para></listitem>
     </varlistentry>
 
     <varlistentry>
     <term><literal>multiPkgs</literal></term>
 
     <listitem><para>Packages to be installed for all architectures supported by
-    a host (i.e. i686 and x86_64 on x86_64 installations).</para></listitem>
+    a host (i.e. i686 and x86_64 on x86_64 installations). Only libraries are
+    installed by default.</para></listitem>
     </varlistentry>
 
     <varlistentry>
@@ -240,30 +232,34 @@ c = lib.makeOverridable f { a = 1; b = 2; }</programlisting>
     <varlistentry>
     <term><literal>extraBuildCommandsMulti</literal></term>
 
-    <listitem><para>Like <literal>extraBuildCommandsMulti</literal>, but
+    <listitem><para>Like <literal>extraBuildCommands</literal>, but
     executed only on multilib architectures.</para></listitem>
     </varlistentry>
+
+    <varlistentry>
+    <term><literal>extraOutputsToInstall</literal></term>
+
+    <listitem><para>Additional derivation outputs to be linked for both
+    target and multi-architecture packages.</para></listitem>
+    </varlistentry>
+
+    <varlistentry>
+    <term><literal>extraInstallCommands</literal></term>
+
+    <listitem><para>Additional commands to be executed for finalizing the
+    derivation with runner script.</para></listitem>
+    </varlistentry>
+
+    <varlistentry>
+    <term><literal>runScript</literal></term>
+
+    <listitem><para>A command that would be executed inside the sandbox and
+    passed all the command line arguments. It defaults to
+    <literal>bash</literal>.</para></listitem>
+    </varlistentry>
   </variablelist>
 
   <para>
-    Additionally, <function>buildFHSUserEnv</function> accepts
-    <literal>runScript</literal> parameter, which is a command that would be
-    executed inside the sandbox and passed all the command line arguments. It
-    default to <literal>bash</literal>.
-  </para>
-  <para>
-    It also uses <literal>CHROOTENV_EXTRA_BINDS</literal> environment variable
-    for binding extra directories in the sandbox to outside places. The format of
-    the variable is <literal>/mnt=test-mnt:/data</literal>, where
-    <literal>/mnt</literal> would be mounted as <literal>/test-mnt</literal>
-    and <literal>/data</literal> would be mounted as <literal>/data</literal>.
-    <literal>extraBindMounts</literal> array argument to
-    <function>buildFHSUserEnv</function> function is prepended to this variable.
-    Latter entries take priority if defined several times -- i.e. in case of
-    <literal>/data=data1:/data=data2</literal> the actual bind path would be
-    <literal>/data2</literal>.
-  </para>
-  <para>
     One can create a simple environment using a <literal>shell.nix</literal>
     like that:
   </para>
diff --git a/doc/languages-frameworks/go.xml b/doc/languages-frameworks/go.xml
index d715765f6a1..7365f5abe68 100644
--- a/doc/languages-frameworks/go.xml
+++ b/doc/languages-frameworks/go.xml
@@ -5,27 +5,29 @@
 <title>Go</title>
 
 <para>The function <varname>buildGoPackage</varname> builds
-standard Go packages.
+standard Go programs.
 </para>
 
 <example xml:id='ex-buildGoPackage'><title>buildGoPackage</title>
 <programlisting>
-net = buildGoPackage rec {
-  name = "go.net-${rev}";
-  goPackagePath = "golang.org/x/net"; <co xml:id='ex-buildGoPackage-1' />
-  subPackages = [ "ipv4" "ipv6" ]; <co xml:id='ex-buildGoPackage-2' />
-  rev = "e0403b4e005";
+deis = buildGoPackage rec {
+  name = "deis-${version}";
+  version = "1.13.0";
+  
+  goPackagePath = "github.com/deis/deis"; <co xml:id='ex-buildGoPackage-1' />
+  subPackages = [ "client" ]; <co xml:id='ex-buildGoPackage-2' />
+
   src = fetchFromGitHub {
-    inherit rev;
-    owner = "golang";
-    repo = "net";
-    sha256 = "1g7cjzw4g4301a3yqpbk8n1d4s97sfby2aysl275x04g0zh8jxqp";
+    owner = "deis";
+    repo = "deis";
+    rev = "v${version}";
+    sha256 = "1qv9lxqx7m18029lj8cw3k7jngvxs4iciwrypdy0gd2nnghc68sw";
   };
-  goPackageAliases = [ "code.google.com/p/go.net" ]; <co xml:id='ex-buildGoPackage-3' />
-  propagatedBuildInputs = [ goPackages.text ]; <co xml:id='ex-buildGoPackage-4' />
-  buildFlags = "--tags release"; <co xml:id='ex-buildGoPackage-5' />
-  disabled = isGo13;<co xml:id='ex-buildGoPackage-6' />
-};
+
+  goDeps = ./deps.json; <co xml:id='ex-buildGoPackage-3' />
+
+  buildFlags = "--tags release"; <co xml:id='ex-buildGoPackage-4' />
+}
 </programlisting>
 </example>
 
@@ -47,50 +49,90 @@ the following arguments are of special significance to the function:
       packages will be built.
     </para>
     <para>
-      In this example only <literal>code.google.com/p/go.net/ipv4</literal> and
-      <literal>code.google.com/p/go.net/ipv6</literal> will be built.
+      In this example only <literal>github.com/deis/deis/client</literal> will be built.
     </para>
   </callout>
 
   <callout arearefs='ex-buildGoPackage-3'>
     <para>
-      <varname>goPackageAliases</varname> is a list of alternative import paths
-      that are valid for this library.
-      Packages that depend on this library will automatically rename
-      import paths that match any of the aliases to <literal>goPackagePath</literal>.
+      <varname>goDeps</varname> is where the Go dependencies of a Go program are listed
+      in a JSON format described below.
     </para>
+  </callout>
+
+  <callout arearefs='ex-buildGoPackage-4'>
     <para>
-      In this example imports will be renamed from
-      <literal>code.google.com/p/go.net</literal> to
-      <literal>golang.org/x/net</literal> in every package that depend on the
-      <literal>go.net</literal> library.
+      <varname>buildFlags</varname> is a list of flags passed to the go build command.
     </para>
   </callout>
 
-  <callout arearefs='ex-buildGoPackage-4'>
+</calloutlist>
+
+</para>
+
+<para>The <varname>goDeps</varname> attribute should point to a JSON file that defines which Go libraries
+  are needed and should be included in <varname>GOPATH</varname> for <varname>buildPhase</varname>.
+
+</para>
+
+<example xml:id='ex-goDeps'><title>deps.json</title>
+<programlisting>
+[ <co xml:id='ex-goDeps-1' />
+    {
+        "goPackagePath": "gopkg.in/yaml.v2", <co xml:id='ex-goDeps-2' />
+        "fetch": {
+          "type": "git", <co xml:id='ex-goDeps-3' />
+          "url": "https://gopkg.in/yaml.v2",
+          "rev": "a83829b6f1293c91addabc89d0571c246397bbf4",
+          "sha256": "1m4dsmk90sbi17571h6pld44zxz7jc4lrnl4f27dpd1l8g5xvjhh"
+        }
+    },
+    {
+    "include": "../../libs.json", <co xml:id='ex-goDeps-4' />
+    "packages": [ <co xml:id='ex-goDeps-5' />
+            "github.com/docopt/docopt-go",
+            "golang.org/x/crypto",
+        ]
+    }
+]
+</programlisting>
+</example>
+
+<para>
+
+<calloutlist>
+
+  <callout arearefs='ex-goDeps-1'>
     <para>
-      <varname>propagatedBuildInputs</varname> is where the dependencies of a Go library are
-      listed. Only libraries should list <varname>propagatedBuildInputs</varname>. If a standalone
-      program is being built instead, use <varname>buildInputs</varname>. If a library's tests require
-      additional dependencies that are not propagated, they should be listed in <varname>buildInputs</varname>.
+      <varname>goDeps</varname> is a list of Go dependencies.
     </para>
   </callout>
 
-  <callout arearefs='ex-buildGoPackage-5'>
+  <callout arearefs='ex-goDeps-2'>
     <para>
-      <varname>buildFlags</varname> is a list of flags passed to the go build command.
+      <varname>goPackagePath</varname> specifies Go package import path.
     </para>
   </callout>
 
-  <callout arearefs='ex-buildGoPackage-6'>
+  <callout arearefs='ex-goDeps-3'>
     <para>
-      If <varname>disabled</varname> is <literal>true</literal>,
-      nix will refuse to build this package.
+      <varname>fetch type</varname> that needs to be used to get package source. If <varname>git</varname>
+      is used there should be <varname>url</varname>, <varname>rev</varname> and <varname>sha256</varname>
+      defined next to it.
     </para>
+  </callout>
+
+  <callout arearefs='ex-goDeps-4'>
     <para>
-      In this example the package will not be built for go 1.3. The <literal>isGo13</literal>
-      is an utility function that returns <literal>true</literal> if go used to build the
-      package has version 1.3.x.
+      <varname>include</varname> could be used to reuse <varname>goDeps</varname> between Go programs.
+      There is a common libs set in <varname>&lt;nixpkgs/pkgs/development/go-modules/libs.json&gt;</varname>
+      with pinned versions of many packages that you can reuse.
+    </para>
+  </callout>
+
+  <callout arearefs='ex-goDeps-5'>
+    <para>
+      <varname>packages</varname> enumerates all Go packages that will be imported from included file.
     </para>
   </callout>
 
@@ -99,12 +141,21 @@ the following arguments are of special significance to the function:
 </para>
 
 <para>
-Reusable Go libraries may be found in the <varname>goPackages</varname> set. You can test
-build a Go package as follows:
+  <varname>buildGoPackage</varname> produces <xref linkend='chap-multiple-output' xrefstyle="select: title" />
+  where <varname>bin</varname> includes program binaries. You can test build a Go binary as follows:
 
-<screen>
-$ nix-build -A goPackages.net
-</screen>
+  <screen>
+    $ nix-build -A deis.bin
+  </screen>
+
+  or build all outputs with:
+
+  <screen>
+    $ nix-build -A deis.all
+  </screen>
+
+  <varname>bin</varname> output will be installed by default with <varname>nix-env -i</varname>
+  or <varname>systemPackages</varname>.
 
 </para>
 
@@ -119,6 +170,7 @@ done
 </screen>
 </para>
 
-  <para>To extract dependency information from a Go package in automated way use <link xlink:href="https://github.com/kamilchm/go2nix">go2nix</link>.</para>
+<para>To extract dependency information from a Go package in automated way use <link xlink:href="https://github.com/kamilchm/go2nix">go2nix</link>.
+  It can produce complete derivation and <varname>goDeps</varname> file for Go programs.</para>
 </section>
 
diff --git a/doc/languages-frameworks/haskell.md b/doc/languages-frameworks/haskell.md
index e066ad110be..3da13dd9ed6 100644
--- a/doc/languages-frameworks/haskell.md
+++ b/doc/languages-frameworks/haskell.md
@@ -378,6 +378,23 @@ special options turned on:
 	  buildInputs = [ R zeromq zlib ];
     }
 
+You can select a particular GHC version to compile with by setting the
+`ghc` attribute as an argument to `buildStackProject`. Better yet, let
+Stack choose what GHC version it wants based on the snapshot specified
+in `stack.yaml` (only works with Stack >= 1.1.3):
+
+    {nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}
+
+    with nixpkgs;
+
+    let R = pkgs.R.override { enableStrictBarrier = true; };
+    in
+    haskell.lib.buildStackProject {
+      name = "HaskellR";
+      buildInputs = [ R zeromq zlib ];
+      inherit ghc;
+    }
+
 [stack-nix-doc]: http://docs.haskellstack.org/en/stable/nix_integration.html
 
 ### How to create ad hoc environments for `nix-shell`
@@ -636,7 +653,7 @@ then you have to download and re-install `foo` and all its dependents from
 scratch:
 
     # nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \
-      | xargs -L 1 nix-store --repair-path --option binary-caches http://hydra.nixos.org
+      | xargs -L 1 nix-store --repair-path
 
 If you're using additional Hydra servers other than `hydra.nixos.org`, then it
 might be necessary to purge the local caches that store data from those
diff --git a/doc/languages-frameworks/python.md b/doc/languages-frameworks/python.md
index dd9af4e827e..99fb6774b82 100644
--- a/doc/languages-frameworks/python.md
+++ b/doc/languages-frameworks/python.md
@@ -532,6 +532,7 @@ All parameters from `mkDerivation` function are still supported.
 * `makeWrapperArgs`: A list of strings. Arguments to be passed to `makeWrapper`, which wraps generated binaries. By default, the arguments to `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`.
 * `installFlags`: A list of strings. Arguments to be passed to `pip install`. To pass options to `python setup.py install`, use `--install-option`. E.g., `installFlags=["--install-option='--cpp_implementation'"].
 * `format`: Format of the source. Options are `setup` for when the source has a `setup.py` and `setuptools` is used to build a wheel, and `wheel` in case the source is already a binary wheel. The default value is `setup`.
+* `catchConflicts` If `true`, abort package build if a package name appears more than once in dependency tree. Default is `true`.
 
 #### `buildPythonApplication` function
 
@@ -648,6 +649,56 @@ community to help save time. No tool is preferred at the moment.
 
 ## FAQ
 
+### How can I install a working Python environment?
+
+As explained in the user's guide installing individual Python packages
+imperatively with `nix-env -i` or declaratively in `environment.systemPackages`
+is not supported. However, it is possible to install a Python environment with packages (`python.buildEnv`).
+
+In the following examples we create an environment with Python 3.5, `numpy` and `ipython`.
+As you might imagine there is one limitation here, and that's you can install
+only one environment at a time. You will notice the complaints about collisions
+when you try to install a second environment.
+
+#### Environment defined in separate `.nix` file
+
+Create a file, e.g. `build.nix`, with the following expression
+```nix
+with import <nixpkgs> {};
+with python35Packages;
+
+python.withPackages (ps: with ps; [ numpy ipython ])
+```
+and install it in your profile with
+```
+nix-env -if build.nix
+```
+Now you can use the Python interpreter, as well as the extra packages that you added to the environment.
+
+#### Environment defined in `~/.nixpkgs/config.nix`
+
+If you prefer to, you could also add the environment as a package override to the Nixpkgs set.
+```
+  packageOverrides = pkgs: with pkgs; with python35Packages; {
+    myEnv = python.withPackages (ps: with ps; [ numpy ipython ]);
+  };
+```
+and install it in your profile with
+```
+nix-env -iA nixos.blogEnv
+```
+Note that I'm using the attribute path here.
+
+#### Environment defined in `/etc/nixos/configuration.nix`
+
+For the sake of completeness, here's another example how to install the environment system-wide.
+
+```nix
+environment.systemPackages = with pkgs; [
+  (python35Packages.python.withPackages (ps: callPackage ../packages/common-python-packages.nix { pythonPackages = ps; }))
+];
+```
+
 ### How to solve circular dependencies?
 
 Consider the packages `A` and `B` that depend on each other. When packaging `B`,
diff --git a/doc/stdenv.xml b/doc/stdenv.xml
index 434b61fd6a4..303ad2db8a7 100644
--- a/doc/stdenv.xml
+++ b/doc/stdenv.xml
@@ -1196,10 +1196,24 @@ echo @foo@
     <term><function>stripHash</function>
     <replaceable>path</replaceable></term>
     <listitem><para>Strips the directory and hash part of a store
-    path, and prints (on standard output) only the name part.  For
-    instance, <literal>stripHash
-    /nix/store/68afga4khv0w...-coreutils-6.12</literal> print
-    <literal>coreutils-6.12</literal>.</para></listitem>
+    path, storing the name part in the environment variable
+    <literal>strippedName</literal>. For example:
+    
+<programlisting>
+stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"
+# prints coreutils-8.24
+echo $strippedName
+</programlisting>
+
+    If you wish to store the result in another variable, then the
+    following idiom may be useful:
+    
+<programlisting>
+name="/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"
+someVar=$(stripHash $name; echo $strippedName)
+</programlisting>
+
+    </para></listitem>
   </varlistentry>