diff options
Diffstat (limited to 'doc/languages-frameworks/python.section.md')
-rw-r--r-- | doc/languages-frameworks/python.section.md | 1682 |
1 files changed, 1682 insertions, 0 deletions
diff --git a/doc/languages-frameworks/python.section.md b/doc/languages-frameworks/python.section.md new file mode 100644 index 00000000000..693ea016e0a --- /dev/null +++ b/doc/languages-frameworks/python.section.md @@ -0,0 +1,1682 @@ +# Python {#python} + +## User Guide {#user-guide} + +### Using Python {#using-python} + +#### Overview {#overview} + +Several versions of the Python interpreter are available on Nix, as well as a +high amount of packages. The attribute `python3` refers to the default +interpreter, which is currently CPython 3.9. The attribute `python` refers to +CPython 2.7 for backwards-compatibility. It is also possible to refer to +specific versions, e.g. `python38` refers to CPython 3.8, and `pypy` refers to +the default PyPy interpreter. + +Python is used a lot, and in different ways. This affects also how it is +packaged. In the case of Python on Nix, an important distinction is made between +whether the package is considered primarily an application, or whether it should +be used as a library, i.e., of primary interest are the modules in +`site-packages` that should be importable. + +In the Nixpkgs tree Python applications can be found throughout, depending on +what they do, and are called from the main package set. Python libraries, +however, are in separate sets, with one set per interpreter version. + +The interpreters have several common attributes. One of these attributes is +`pkgs`, which is a package set of Python libraries for this specific +interpreter. E.g., the `toolz` package corresponding to the default interpreter +is `python.pkgs.toolz`, and the CPython 3.8 version is `python38.pkgs.toolz`. +The main package set contains aliases to these package sets, e.g. +`pythonPackages` refers to `python.pkgs` and `python38Packages` to +`python38.pkgs`. + +#### Installing Python and packages {#installing-python-and-packages} + +The Nix and NixOS manuals explain how packages are generally installed. In the +case of Python and Nix, it is important to make a distinction between whether the +package is considered an application or a library. + +Applications on Nix are typically installed into your user profile imperatively +using `nix-env -i`, and on NixOS declaratively by adding the package name to +`environment.systemPackages` in `/etc/nixos/configuration.nix`. Dependencies +such as libraries are automatically installed and should not be installed +explicitly. + +The same goes for Python applications. Python applications can be installed in +your profile, and will be wrapped to find their exact library dependencies, +without impacting other applications or polluting your user environment. + +But Python libraries you would like to use for development cannot be installed, +at least not individually, because they won't be able to find each other +resulting in import errors. Instead, it is possible to create an environment +with `python.buildEnv` or `python.withPackages` where the interpreter and other +executables are wrapped to be able to find each other and all of the modules. + +In the following examples we will start by creating a simple, ad-hoc environment +with a nix-shell that has `numpy` and `toolz` in Python 3.8; then we will create +a re-usable environment in a single-file Python script; then we will create a +full Python environment for development with this same environment. + +Philosphically, this should be familiar to users who are used to a `venv` style +of development: individual projects create their own Python environments without +impacting the global environment or each other. + +#### Ad-hoc temporary Python environment with `nix-shell` {#ad-hoc-temporary-python-environment-with-nix-shell} + +The simplest way to start playing with the way nix wraps and sets up Python +environments is with `nix-shell` at the cmdline. These environments create a +temporary shell session with a Python and a *precise* list of packages (plus +their runtime dependencies), with no other Python packages in the Python +interpreter's scope. + +To create a Python 3.8 session with `numpy` and `toolz` available, run: + +```sh +$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' +``` + +By default `nix-shell` will start a `bash` session with this interpreter in our +`PATH`, so if we then run: + +```Python console +[nix-shell:~/src/nixpkgs]$ python3 +Python 3.8.1 (default, Dec 18 2019, 19:06:26) +[GCC 9.2.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import numpy; import toolz +``` + +Note that no other modules are in scope, even if they were imperatively +installed into our user environment as a dependency of a Python application: + +```Python console +>>> import requests +Traceback (most recent call last): + File "<stdin>", line 1, in <module> +ModuleNotFoundError: No module named 'requests' +``` + +We can add as many additional modules onto the `nix-shell` as we need, and we +will still get 1 wrapped Python interpreter. We can start the interpreter +directly like so: + +```sh +$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz requests ])' --run python3 +these derivations will be built: + /nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv +building '/nix/store/xbdsrqrsfa1yva5s7pzsra8k08gxlbz1-python3-3.8.1-env.drv'... +created 277 symlinks in user environment +Python 3.8.1 (default, Dec 18 2019, 19:06:26) +[GCC 9.2.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import requests +>>> +``` + +Notice that this time it built a new Python environment, which now includes +`requests`. Building an environment just creates wrapper scripts that expose the +selected dependencies to the interpreter while re-using the actual modules. This +means if any other env has installed `requests` or `numpy` in a different +context, we don't need to recompile them -- we just recompile the wrapper script +that sets up an interpreter pointing to them. This matters much more for "big" +modules like `pytorch` or `tensorflow`. + +Module names usually match their names on [pypi.org](https://pypi.org/), but +you can use the [Nixpkgs search website](https://nixos.org/nixos/packages.html) +to find them as well (along with non-python packages). + +At this point we can create throwaway experimental Python environments with +arbitrary dependencies. This is a good way to get a feel for how the Python +interpreter and dependencies work in Nix and NixOS, but to do some actual +development, we'll want to make it a bit more persistent. + +##### Running Python scripts and using `nix-shell` as shebang {#running-python-scripts-and-using-nix-shell-as-shebang} + +Sometimes, we have a script whose header looks like this: + +```python +#!/usr/bin/env python3 +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +Executing this script requires a `python3` that has `numpy`. Using what we learned +in the previous section, we could startup a shell and just run it like so: + +```ShellSession +$ nix-shell -p 'python38.withPackages(ps: with ps; [ numpy ])' --run 'python3 foo.py' +The dot product of [1 2] and [3 4] is: 11 +``` + +But if we maintain the script ourselves, and if there are more dependencies, it +may be nice to encode those dependencies in source to make the script re-usable +without that bit of knowledge. That can be done by using `nix-shell` as a +[shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)), like so: + +```python +#!/usr/bin/env nix-shell +#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])" +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +Then we simply execute it, without requiring any environment setup at all! + +```sh +$ ./foo.py +The dot product of [1 2] and [3 4] is: 11 +``` + +If the dependencies are not available on the host where `foo.py` is executed, it +will build or download them from a Nix binary cache prior to starting up, prior +that it is executed on a machine with a multi-user nix installation. + +This provides a way to ship a self bootstrapping Python script, akin to a +statically linked binary, where it can be run on any machine (provided nix is +installed) without having to assume that `numpy` is installed globally on the +system. + +By default it is pulling the import checkout of Nixpkgs itself from our nix +channel, which is nice as it cache aligns with our other package builds, but we +can make it fully reproducible by pinning the `nixpkgs` import: + +```python +#!/usr/bin/env nix-shell +#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.numpy ])" +#!nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/d373d80b1207d52621961b16aa4a3438e4f98167.tar.gz +import numpy as np +a = np.array([1,2]) +b = np.array([3,4]) +print(f"The dot product of {a} and {b} is: {np.dot(a, b)}") +``` + +This will execute with the exact same versions of Python 3.8, numpy, and system +dependencies a year from now as it does today, because it will always use +exactly git commit `d373d80b1207d52621961b16aa4a3438e4f98167` of Nixpkgs for all +of the package versions. + +This is also a great way to ensure the script executes identically on different +servers. + +##### Load environment from `.nix` expression {#load-environment-from-.nix-expression} + +We've now seen how to create an ad-hoc temporary shell session, and how to +create a single script with Python dependencies, but in the course of normal +development we're usually working in an entire package repository. + +As explained in the Nix manual, `nix-shell` can also load an expression from a +`.nix` file. Say we want to have Python 3.8, `numpy` and `toolz`, like before, +in an environment. We can add a `shell.nix` file describing our dependencies: + +```nix +with import <nixpkgs> {}; +(python38.withPackages (ps: [ps.numpy ps.toolz])).env +``` + +And then at the command line, just typing `nix-shell` produces the same +environment as before. In a normal project, we'll likely have many more +dependencies; this can provide a way for developers to share the environments +with each other and with CI builders. + +What's happening here? + +1. We begin with importing the Nix Packages collections. `import <nixpkgs>` + imports the `<nixpkgs>` function, `{}` calls it and the `with` statement + brings all attributes of `nixpkgs` in the local scope. These attributes form + the main package set. +2. Then we create a Python 3.8 environment with the `withPackages` function, as before. +3. The `withPackages` function expects us to provide a function as an argument + that takes the set of all Python packages and returns a list of packages to + include in the environment. Here, we select the packages `numpy` and `toolz` + from the package set. + +To combine this with `mkShell` you can: + +```nix +with import <nixpkgs> {}; +let + pythonEnv = python38.withPackages (ps: [ + ps.numpy + ps.toolz + ]); +in mkShell { + packages = [ + pythonEnv + + black + mypy + + libffi + openssl + ]; +} +``` + +This will create a unified environment that has not just our Python interpreter +and its Python dependencies, but also tools like `black` or `mypy` and libraries +like `libffi` the `openssl` in scope. This is generic and can span any number of +tools or languages across the Nixpkgs ecosystem. + +##### Installing environments globally on the system {#installing-environments-globally-on-the-system} + +Up to now, we've been creating environments scoped to an ad-hoc shell session, +or a single script, or a single project. This is generally advisable, as it +avoids pollution across contexts. + +However, sometimes we know we will often want a Python with some basic packages, +and want this available without having to enter into a shell or build context. +This can be useful to have things like vim/emacs editors and plugins or shell +tools "just work" without having to set them up, or when running other software +that expects packages to be installed globally. + +To create your own custom environment, create a file in `~/.config/nixpkgs/overlays/` +that looks like this: + +```nix +# ~/.config/nixpkgs/overlays/myEnv.nix +self: super: { + myEnv = super.buildEnv { + name = "myEnv"; + paths = [ + # A Python 3 interpreter with some packages + (self.python3.withPackages ( + ps: with ps; [ + pyflakes + pytest + python-language-server + ] + )) + + # Some other packages we'd like as part of this env + self.mypy + self.black + self.ripgrep + self.tmux + ]; + }; +} +``` + +You can then build and install this to your profile with: + +```sh +nix-env -iA myEnv +``` + +One limitation of this is that you can only have 1 Python env installed +globally, since they conflict on the `python` to load out of your `PATH`. + +If you get a conflict or prefer to keep the setup clean, you can have `nix-env` +atomically *uninstall* all other imperatively installed packages and replace +your profile with just `myEnv` by using the `--replace` flag. + +##### Environment defined in `/etc/nixos/configuration.nix` {#environment-defined-in-etcnixosconfiguration.nix} + +For the sake of completeness, here's how to install the environment system-wide +on NixOS. + +```nix +{ # ... + + environment.systemPackages = with pkgs; [ + (python38.withPackages(ps: with ps; [ numpy toolz ])) + ]; +} +``` + +### Developing with Python {#developing-with-python} + +Above, we were mostly just focused on use cases and what to do to get started +creating working Python environments in nix. + +Now that you know the basics to be up and running, it is time to take a step +back and take a deeper look at how Python packages are packaged on Nix. Then, +we will look at how you can use development mode with your code. + +#### Python library packages in Nixpkgs {#python-library-packages-in-nixpkgs} + +With Nix all packages are built by functions. The main function in Nix for +building Python libraries is `buildPythonPackage`. Let's see how we can build the +`toolz` package. + +```nix +{ lib, buildPythonPackage, fetchPypi }: + +buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + doCheck = false; + + meta = with lib; { + homepage = "https://github.com/pytoolz/toolz"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +What happens here? The function `buildPythonPackage` is called and as argument +it accepts a set. In this case the set is a recursive set, `rec`. One of the +arguments is the name of the package, which consists of a basename (generally +following the name on PyPi) and a version. Another argument, `src` specifies the +source, which in this case is fetched from PyPI using the helper function +`fetchPypi`. The argument `doCheck` is used to set whether tests should be run +when building the package. Furthermore, we specify some (optional) meta +information. The output of the function is a derivation. + +An expression for `toolz` can be found in the Nixpkgs repository. As explained +in the introduction of this Python section, a derivation of `toolz` is available +for each interpreter version, e.g. `python38.pkgs.toolz` refers to the `toolz` +derivation corresponding to the CPython 3.8 interpreter. + +The above example works when you're directly working on +`pkgs/top-level/python-packages.nix` in the Nixpkgs repository. Often though, +you will want to test a Nix expression outside of the Nixpkgs tree. + +The following expression creates a derivation for the `toolz` package, +and adds it along with a `numpy` package to a Python environment. + +```nix +with import <nixpkgs> {}; + +( let + my_toolz = python38.pkgs.buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = python38.pkgs.fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + doCheck = false; + + meta = { + homepage = "https://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + }; + }; + + in python38.withPackages (ps: [ps.numpy my_toolz]) +).env +``` + +Executing `nix-shell` will result in an environment in which you can use +Python 3.8 and the `toolz` package. As you can see we had to explicitly mention +for which Python version we want to build a package. + +So, what did we do here? Well, we took the Nix expression that we used earlier +to build a Python environment, and said that we wanted to include our own +version of `toolz`, named `my_toolz`. To introduce our own package in the scope +of `withPackages` we used a `let` expression. You can see that we used +`ps.numpy` to select numpy from the nixpkgs package set (`ps`). We did not take +`toolz` from the Nixpkgs package set this time, but instead took our own version +that we introduced with the `let` expression. + +#### Handling dependencies {#handling-dependencies} + +Our example, `toolz`, does not have any dependencies on other Python packages or +system libraries. According to the manual, `buildPythonPackage` uses the +arguments `buildInputs` and `propagatedBuildInputs` to specify dependencies. If +something is exclusively a build-time dependency, then the dependency should be +included in `buildInputs`, but if it is (also) a runtime dependency, then it +should be added to `propagatedBuildInputs`. Test dependencies are considered +build-time dependencies and passed to `checkInputs`. + +The following example shows which arguments are given to `buildPythonPackage` in +order to build [`datashape`](https://github.com/blaze/datashape). + +```nix +{ lib, buildPythonPackage, fetchPypi, numpy, multipledispatch, python-dateutil, pytest }: + +buildPythonPackage rec { + pname = "datashape"; + version = "0.4.7"; + + src = fetchPypi { + inherit pname version; + sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278"; + }; + + checkInputs = [ pytest ]; + propagatedBuildInputs = [ numpy multipledispatch python-dateutil ]; + + meta = with lib; { + homepage = "https://github.com/ContinuumIO/datashape"; + description = "A data description language"; + license = licenses.bsd2; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +We can see several runtime dependencies, `numpy`, `multipledispatch`, and +`python-dateutil`. Furthermore, we have one `checkInputs`, i.e. `pytest`. `pytest` is a +test runner and is only used during the `checkPhase` and is therefore not added +to `propagatedBuildInputs`. + +In the previous case we had only dependencies on other Python packages to consider. +Occasionally you have also system libraries to consider. E.g., `lxml` provides +Python bindings to `libxml2` and `libxslt`. These libraries are only required +when building the bindings and are therefore added as `buildInputs`. + +```nix +{ lib, pkgs, buildPythonPackage, fetchPypi }: + +buildPythonPackage rec { + pname = "lxml"; + version = "3.4.4"; + + src = fetchPypi { + inherit pname version; + sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk"; + }; + + buildInputs = [ pkgs.libxml2 pkgs.libxslt ]; + + meta = with lib; { + description = "Pythonic binding for the libxml2 and libxslt libraries"; + homepage = "https://lxml.de"; + license = licenses.bsd3; + maintainers = with maintainers; [ sjourdois ]; + }; +} +``` + +In this example `lxml` and Nix are able to work out exactly where the relevant +files of the dependencies are. This is not always the case. + +The example below shows bindings to The Fastest Fourier Transform in the West, +commonly known as FFTW. On Nix we have separate packages of FFTW for the +different types of floats (`"single"`, `"double"`, `"long-double"`). The +bindings need all three types, and therefore we add all three as `buildInputs`. +The bindings don't expect to find each of them in a different folder, and +therefore we have to set `LDFLAGS` and `CFLAGS`. + +```nix +{ lib, pkgs, buildPythonPackage, fetchPypi, numpy, scipy }: + +buildPythonPackage rec { + pname = "pyFFTW"; + version = "0.9.2"; + + src = fetchPypi { + inherit pname version; + sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074"; + }; + + buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble]; + + propagatedBuildInputs = [ numpy scipy ]; + + # Tests cannot import pyfftw. pyfftw works fine though. + doCheck = false; + + preConfigure = '' + export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib" + export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include" + ''; + + meta = with lib; { + description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms"; + homepage = "http://hgomersall.github.com/pyFFTW"; + license = with licenses; [ bsd2 bsd3 ]; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +Note also the line `doCheck = false;`, we explicitly disabled running the test-suite. + +#### Testing Python Packages {#testing-python-packages} + +It is highly encouraged to have testing as part of the package build. This +helps to avoid situations where the package was able to build and install, +but is not usable at runtime. Currently, all packages will use the `test` +command provided by the setup.py (i.e. `python setup.py test`). However, +this is currently deprecated https://github.com/pypa/setuptools/pull/1878 +and your package should provide its own checkPhase. + +*NOTE:* The `checkPhase` for python maps to the `installCheckPhase` on a +normal derivation. This is due to many python packages not behaving well +to the pre-installed version of the package. Version info, and natively +compiled extensions generally only exist in the install directory, and +thus can cause issues when a test suite asserts on that behavior. + +*NOTE:* Tests should only be disabled if they don't agree with nix +(e.g. external dependencies, network access, flakey tests), however, +as many tests should be enabled as possible. Failing tests can still be +a good indication that the package is not in a valid state. + +#### Using pytest {#using-pytest} + +Pytest is the most common test runner for python repositories. A trivial +test run would be: + +``` + checkInputs = [ pytest ]; + checkPhase = "pytest"; +``` + +However, many repositories' test suites do not translate well to nix's build +sandbox, and will generally need many tests to be disabled. + +To filter tests using pytest, one can do the following: + +``` + checkInputs = [ pytest ]; + # avoid tests which need additional data or touch network + checkPhase = '' + pytest tests/ --ignore=tests/integration -k 'not download and not update' + ''; +``` + +`--ignore` will tell pytest to ignore that file or directory from being +collected as part of a test run. This is useful is a file uses a package +which is not available in nixpkgs, thus skipping that test file is much +easier than having to create a new package. + +`-k` is used to define a predicate for test names. In this example, we are +filtering out tests which contain `download` or `update` in their test case name. +Only one `-k` argument is allowed, and thus a long predicate should be concatenated +with “\\” and wrapped to the next line. + +*NOTE:* In pytest==6.0.1, the use of “\\” to continue a line (e.g. `-k 'not download \'`) has +been removed, in this case, it's recommended to use `pytestCheckHook`. + +#### Using pytestCheckHook {#using-pytestcheckhook} + +`pytestCheckHook` is a convenient hook which will substitute the setuptools +`test` command for a checkPhase which runs `pytest`. This is also beneficial +when a package may need many items disabled to run the test suite. + +Using the example above, the analagous pytestCheckHook usage would be: + +``` + checkInputs = [ pytestCheckHook ]; + + # requires additional data + pytestFlagsArray = [ "tests/" "--ignore=tests/integration" ]; + + disabledTests = [ + # touches network + "download" + "update" + ]; + + disabledTestPaths = [ + "tests/test_failing.py" + ]; +``` + +This is expecially useful when tests need to be conditionallydisabled, +for example: + +``` + disabledTests = [ + # touches network + "download" + "update" + ] ++ lib.optionals (pythonAtLeast "3.8") [ + # broken due to python3.8 async changes + "async" + ] ++ lib.optionals stdenv.isDarwin [ + # can fail when building with other packages + "socket" + ]; +``` +Trying to concatenate the related strings to disable tests in a regular checkPhase +would be much harder to read. This also enables us to comment on why specific tests +are disabled. + +#### Using pythonImportsCheck {#using-pythonimportscheck} + +Although unit tests are highly prefered to validate correctness of a package, not +all packages have test suites that can be ran easily, and some have none at all. +To help ensure the package still works, `pythonImportsCheck` can attempt to import +the listed modules. + +``` + pythonImportsCheck = [ "requests" "urllib" ]; +``` +roughly translates to: +``` + postCheck = '' + PYTHONPATH=$out/${python.sitePackages}:$PYTHONPATH + python -c "import requests; import urllib" + ''; +``` +However, this is done in it's own phase, and not dependent on whether `doCheck = true;` + +This can also be useful in verifying that the package doesn't assume commonly +present packages (e.g. `setuptools`) + +### Develop local package {#develop-local-package} + +As a Python developer you're likely aware of [development mode](http://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) +(`python setup.py develop`); instead of installing the package this command +creates a special link to the project code. That way, you can run updated code +without having to reinstall after each and every change you make. Development +mode is also available. Let's see how you can use it. + +In the previous Nix expression the source was fetched from an url. We can also +refer to a local source instead using `src = ./path/to/source/tree;` + +If we create a `shell.nix` file which calls `buildPythonPackage`, and if `src` +is a local source, and if the local source has a `setup.py`, then development +mode is activated. + +In the following example we create a simple environment that has a Python 3.8 +version of our package in it, as well as its dependencies and other packages we +like to have in the environment, all specified with `propagatedBuildInputs`. +Indeed, we can just add any package we like to have in our environment to +`propagatedBuildInputs`. + +```nix +with import <nixpkgs> {}; +with python38Packages; + +buildPythonPackage rec { + name = "mypackage"; + src = ./path/to/package/source; + propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ]; +} +``` + +It is important to note that due to how development mode is implemented on Nix +it is not possible to have multiple packages simultaneously in development mode. + +### Organising your packages {#organising-your-packages} + +So far we discussed how you can use Python on Nix, and how you can develop with +it. We've looked at how you write expressions to package Python packages, and we +looked at how you can create environments in which specified packages are +available. + +At some point you'll likely have multiple packages which you would +like to be able to use in different projects. In order to minimise unnecessary +duplication we now look at how you can maintain a repository with your +own packages. The important functions here are `import` and `callPackage`. + +### Including a derivation using `callPackage` {#including-a-derivation-using-callpackage} + +Earlier we created a Python environment using `withPackages`, and included the +`toolz` package via a `let` expression. +Let's split the package definition from the environment definition. + +We first create a function that builds `toolz` in `~/path/to/toolz/release.nix` + +```nix +{ lib, buildPythonPackage }: + +buildPythonPackage rec { + pname = "toolz"; + version = "0.10.0"; + + src = fetchPypi { + inherit pname version; + sha256 = "08fdd5ef7c96480ad11c12d472de21acd32359996f69a5259299b540feba4560"; + }; + + meta = with lib; { + homepage = "https://github.com/pytoolz/toolz/"; + description = "List processing tools and functional utilities"; + license = licenses.bsd3; + maintainers = with maintainers; [ fridh ]; + }; +} +``` + +It takes an argument `buildPythonPackage`. We now call this function using +`callPackage` in the definition of our environment + +```nix +with import <nixpkgs> {}; + +( let + toolz = callPackage /path/to/toolz/release.nix { + buildPythonPackage = python38Packages.buildPythonPackage; + }; + in python38.withPackages (ps: [ ps.numpy toolz ]) +).env +``` + +Important to remember is that the Python version for which the package is made +depends on the `python` derivation that is passed to `buildPythonPackage`. Nix +tries to automatically pass arguments when possible, which is why generally you +don't explicitly define which `python` derivation should be used. In the above +example we use `buildPythonPackage` that is part of the set `python38Packages`, +and in this case the `python38` interpreter is automatically used. + +## Reference {#reference} + +### Interpreters {#interpreters} + +Versions 2.7, 3.7, 3.8 and 3.9 of the CPython interpreter are available as +respectively `python27`, `python37`, `python38` and `python39`. The +aliases `python2` and `python3` correspond to respectively `python27` and +`python39`. The attribute `python` maps to `python2`. The PyPy interpreters +compatible with Python 2.7 and 3 are available as `pypy27` and `pypy3`, with +aliases `pypy2` mapping to `pypy27` and `pypy` mapping to `pypy2`. The Nix +expressions for the interpreters can be found in +`pkgs/development/interpreters/python`. + +All packages depending on any Python interpreter get appended +`out/{python.sitePackages}` to `$PYTHONPATH` if such directory +exists. + +#### Missing `tkinter` module standard library {#missing-tkinter-module-standard-library} + +To reduce closure size the `Tkinter`/`tkinter` is available as a separate package, `pythonPackages.tkinter`. + +#### Attributes on interpreters packages {#attributes-on-interpreters-packages} + +Each interpreter has the following attributes: + +- `libPrefix`. Name of the folder in `${python}/lib/` for corresponding interpreter. +- `interpreter`. Alias for `${python}/bin/${executable}`. +- `buildEnv`. Function to build python interpreter environments with extra packages bundled together. See section *python.buildEnv function* for usage and documentation. +- `withPackages`. Simpler interface to `buildEnv`. See section *python.withPackages function* for usage and documentation. +- `sitePackages`. Alias for `lib/${libPrefix}/site-packages`. +- `executable`. Name of the interpreter executable, e.g. `python3.8`. +- `pkgs`. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing `packageOverrides`. + +### Optimizations {#optimizations} + +The Python interpreters are by default not build with optimizations enabled, because +the builds are in that case not reproducible. To enable optimizations, override the +interpreter of interest, e.g using + +``` +let + pkgs = import ./. {}; + mypython = pkgs.python3.override { + enableOptimizations = true; + reproducibleBuild = false; + self = mypython; + }; +in mypython +``` + +### Building packages and applications {#building-packages-and-applications} + +Python libraries and applications that use `setuptools` or +`distutils` are typically built with respectively the `buildPythonPackage` and +`buildPythonApplication` functions. These two functions also support installing a `wheel`. + +All Python packages reside in `pkgs/top-level/python-packages.nix` and all +applications elsewhere. In case a package is used as both a library and an +application, then the package should be in `pkgs/top-level/python-packages.nix` +since only those packages are made available for all interpreter versions. The +preferred location for library expressions is in +`pkgs/development/python-modules`. It is important that these packages are +called from `pkgs/top-level/python-packages.nix` and not elsewhere, to guarantee +the right version of the package is built. + +Based on the packages defined in `pkgs/top-level/python-packages.nix` an +attribute set is created for each available Python interpreter. The available +sets are + +* `pkgs.python27Packages` +* `pkgs.python37Packages` +* `pkgs.python38Packages` +* `pkgs.python39Packages` +* `pkgs.python310Packages` +* `pkgs.python311Packages` +* `pkgs.pypyPackages` + +and the aliases + +* `pkgs.python2Packages` pointing to `pkgs.python27Packages` +* `pkgs.python3Packages` pointing to `pkgs.python39Packages` +* `pkgs.pythonPackages` pointing to `pkgs.python2Packages` + +#### `buildPythonPackage` function {#buildpythonpackage-function} + +The `buildPythonPackage` function is implemented in +`pkgs/development/interpreters/python/mk-python-derivation` +using setup hooks. + +The following is an example: + +```nix +{ lib, buildPythonPackage, fetchPypi, hypothesis, setuptools-scm, attrs, py, setuptools, six, pluggy }: + +buildPythonPackage rec { + pname = "pytest"; + version = "3.3.1"; + + src = fetchPypi { + inherit pname version; + sha256 = "cf8436dc59d8695346fcd3ab296de46425ecab00d64096cebe79fb51ecb2eb93"; + }; + + postPatch = '' + # don't test bash builtins + rm testing/test_argcomplete.py + ''; + + checkInputs = [ hypothesis ]; + nativeBuildInputs = [ setuptools-scm ]; + propagatedBuildInputs = [ attrs py setuptools six pluggy ]; + + meta = with lib; { + maintainers = with maintainers; [ domenkozar lovek323 madjar lsix ]; + description = "Framework for writing tests"; + }; +} +``` + +The `buildPythonPackage` mainly does four things: + +* In the `buildPhase`, it calls `${python.interpreter} setup.py bdist_wheel` to + build a wheel binary zipfile. +* In the `installPhase`, it installs the wheel file using `pip install *.whl`. +* In the `postFixup` phase, the `wrapPythonPrograms` bash function is called to + wrap all programs in the `$out/bin/*` directory to include `$PATH` + environment variable and add dependent libraries to script's `sys.path`. +* In the `installCheck` phase, `${python.interpreter} setup.py test` is ran. + +By default tests are run because `doCheck = true`. Test dependencies, like +e.g. the test runner, should be added to `checkInputs`. + +By default `meta.platforms` is set to the same value +as the interpreter unless overridden otherwise. + +##### `buildPythonPackage` parameters {#buildpythonpackage-parameters} + +All parameters from `stdenv.mkDerivation` function are still supported. The +following are specific to `buildPythonPackage`: + +* `catchConflicts ? true`: If `true`, abort package build if a package name + appears more than once in dependency tree. Default is `true`. +* `disabled` ? false: If `true`, package is not built for the particular Python + interpreter version. +* `dontWrapPythonPrograms ? false`: Skip wrapping of Python programs. +* `permitUserSite ? false`: Skip setting the `PYTHONNOUSERSITE` environment + variable in wrapped programs. +* `format ? "setuptools"`: Format of the source. Valid options are + `"setuptools"`, `"pyproject"`, `"flit"`, `"wheel"`, and `"other"`. + `"setuptools"` is for when the source has a `setup.py` and `setuptools` is + used to build a wheel, `flit`, in case `flit` should be used to build a wheel, + and `wheel` in case a wheel is provided. Use `other` when a custom + `buildPhase` and/or `installPhase` is needed. +* `makeWrapperArgs ? []`: A list of strings. Arguments to be passed to + `makeWrapper`, which wraps generated binaries. By default, the arguments to + `makeWrapper` set `PATH` and `PYTHONPATH` environment variables before calling + the binary. Additional arguments here can allow a developer to set environment + variables which will be available when the binary is run. For example, + `makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"]`. +* `namePrefix`: Prepends text to `${name}` parameter. In case of libraries, this + defaults to `"python3.8-"` for Python 3.8, etc., and in case of applications + to `""`. +* `pipInstallFlags ? []`: A list of strings. Arguments to be passed to `pip + install`. To pass options to `python setup.py install`, use + `--install-option`. E.g., `pipInstallFlags=["--install-option='--cpp_implementation'"]`. +* `pythonPath ? []`: List of packages to be added into `$PYTHONPATH`. Packages + in `pythonPath` are not propagated (contrary to `propagatedBuildInputs`). +* `preShellHook`: Hook to execute commands before `shellHook`. +* `postShellHook`: Hook to execute commands after `shellHook`. +* `removeBinByteCode ? true`: Remove bytecode from `/bin`. Bytecode is only + created when the filenames end with `.py`. +* `setupPyGlobalFlags ? []`: List of flags passed to `setup.py` command. +* `setupPyBuildFlags ? []`: List of flags passed to `setup.py build_ext` command. + +The `stdenv.mkDerivation` function accepts various parameters for describing +build inputs (see "Specifying dependencies"). The following are of special +interest for Python packages, either because these are primarily used, or +because their behaviour is different: + +* `nativeBuildInputs ? []`: Build-time only dependencies. Typically executables + as well as the items listed in `setup_requires`. +* `buildInputs ? []`: Build and/or run-time dependencies that need to be + compiled for the host machine. Typically non-Python libraries which are being + linked. +* `checkInputs ? []`: Dependencies needed for running the `checkPhase`. These + are added to `nativeBuildInputs` when `doCheck = true`. Items listed in + `tests_require` go here. +* `propagatedBuildInputs ? []`: Aside from propagating dependencies, + `buildPythonPackage` also injects code into and wraps executables with the + paths included in this list. Items listed in `install_requires` go here. + +##### Overriding Python packages {#overriding-python-packages} + +The `buildPythonPackage` function has a `overridePythonAttrs` method that can be +used to override the package. In the following example we create an environment +where we have the `blaze` package using an older version of `pandas`. We +override first the Python interpreter and pass `packageOverrides` which contains +the overrides for packages in the package set. + +```nix +with import <nixpkgs> {}; + +(let + python = let + packageOverrides = self: super: { + pandas = super.pandas.overridePythonAttrs(old: rec { + version = "0.19.1"; + src = super.fetchPypi { + pname = "pandas"; + inherit version; + sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295"; + }; + }); + }; + in pkgs.python3.override {inherit packageOverrides; self = python;}; + +in python.withPackages(ps: [ps.blaze])).env +``` + +#### Optional extra dependencies + +Some packages define optional dependencies for additional features. With +`setuptools` this is called `extras_require` and `flit` calls it `extras-require`. A +method for supporting this is by declaring the extras of a package in its +`passthru`, e.g. in case of the package `dask` + +```nix +passthru.extras-require = { + complete = [ distributed ]; +}; +``` + +and letting the package requiring the extra add the list to its dependencies + +```nix +propagatedBuildInputs = [ + ... +] ++ dask.extras-require.complete; +``` + +Note this method is preferred over adding parameters to builders, as that can +result in packages depending on different variants and thereby causing +collisions. + +#### `buildPythonApplication` function {#buildpythonapplication-function} + +The `buildPythonApplication` function is practically the same as +`buildPythonPackage`. The main purpose of this function is to build a Python +package where one is interested only in the executables, and not importable +modules. For that reason, when adding this package to a `python.buildEnv`, the +modules won't be made available. + +Another difference is that `buildPythonPackage` by default prefixes the names of +the packages with the version of the interpreter. Because this is irrelevant for +applications, the prefix is omitted. + +When packaging a Python application with `buildPythonApplication`, it should be +called with `callPackage` and passed `python` or `pythonPackages` (possibly +specifying an interpreter version), like this: + +```nix +{ lib, python3 }: + +python3.pkgs.buildPythonApplication rec { + pname = "luigi"; + version = "2.7.9"; + + src = python3.pkgs.fetchPypi { + inherit pname version; + sha256 = "035w8gqql36zlan0xjrzz9j4lh9hs0qrsgnbyw07qs7lnkvbdv9x"; + }; + + propagatedBuildInputs = with python3.pkgs; [ tornado python-daemon ]; + + meta = with lib; { + ... + }; +} +``` + +This is then added to `all-packages.nix` just as any other application would be. + +```nix +luigi = callPackage ../applications/networking/cluster/luigi { }; +``` + +Since the package is an application, a consumer doesn't need to care about +Python versions or modules, which is why they don't go in `pythonPackages`. + +#### `toPythonApplication` function {#topythonapplication-function} + +A distinction is made between applications and libraries, however, sometimes a +package is used as both. In this case the package is added as a library to +`python-packages.nix` and as an application to `all-packages.nix`. To reduce +duplication the `toPythonApplication` can be used to convert a library to an +application. + +The Nix expression shall use `buildPythonPackage` and be called from +`python-packages.nix`. A reference shall be created from `all-packages.nix` to +the attribute in `python-packages.nix`, and the `toPythonApplication` shall be +applied to the reference: + +```nix +youtube-dl = with pythonPackages; toPythonApplication youtube-dl; +``` + +#### `toPythonModule` function {#topythonmodule-function} + +In some cases, such as bindings, a package is created using +`stdenv.mkDerivation` and added as attribute in `all-packages.nix`. The Python +bindings should be made available from `python-packages.nix`. The +`toPythonModule` function takes a derivation and makes certain Python-specific +modifications. + +```nix +opencv = toPythonModule (pkgs.opencv.override { + enablePython = true; + pythonPackages = self; +}); +``` + +Do pay attention to passing in the right Python version! + +#### `python.buildEnv` function {#python.buildenv-function} + +Python environments can be created using the low-level `pkgs.buildEnv` function. +This example shows how to create an environment that has the Pyramid Web Framework. +Saving the following as `default.nix` + +```nix +with import <nixpkgs> {}; + +python.buildEnv.override { + extraLibs = [ pythonPackages.pyramid ]; + ignoreCollisions = true; +} +``` + +and running `nix-build` will create + +``` +/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env +``` + +with wrapped binaries in `bin/`. + +You can also use the `env` attribute to create local environments with needed +packages installed. This is somewhat comparable to `virtualenv`. For example, +running `nix-shell` with the following `shell.nix` + +```nix +with import <nixpkgs> {}; + +(python3.buildEnv.override { + extraLibs = with python3Packages; [ numpy requests ]; +}).env +``` + +will drop you into a shell where Python will have the +specified packages in its path. + +##### `python.buildEnv` arguments {#python.buildenv-arguments} + + +* `extraLibs`: List of packages installed inside the environment. +* `postBuild`: Shell command executed after the build of environment. +* `ignoreCollisions`: Ignore file collisions inside the environment (default is `false`). +* `permitUserSite`: Skip setting the `PYTHONNOUSERSITE` environment variable in + wrapped binaries in the environment. + +#### `python.withPackages` function {#python.withpackages-function} + +The `python.withPackages` function provides a simpler interface to the `python.buildEnv` functionality. +It takes a function as an argument that is passed the set of python packages and returns the list +of the packages to be included in the environment. Using the `withPackages` function, the previous +example for the Pyramid Web Framework environment can be written like this: + +```nix +with import <nixpkgs> {}; + +python.withPackages (ps: [ps.pyramid]) +``` + +`withPackages` passes the correct package set for the specific interpreter +version as an argument to the function. In the above example, `ps` equals +`pythonPackages`. But you can also easily switch to using python3: + +```nix +with import <nixpkgs> {}; + +python3.withPackages (ps: [ps.pyramid]) +``` + +Now, `ps` is set to `python3Packages`, matching the version of the interpreter. + +As `python.withPackages` simply uses `python.buildEnv` under the hood, it also +supports the `env` attribute. The `shell.nix` file from the previous section can +thus be also written like this: + +```nix +with import <nixpkgs> {}; + +(python38.withPackages (ps: [ps.numpy ps.requests])).env +``` + +In contrast to `python.buildEnv`, `python.withPackages` does not support the +more advanced options such as `ignoreCollisions = true` or `postBuild`. If you +need them, you have to use `python.buildEnv`. + +Python 2 namespace packages may provide `__init__.py` that collide. In that case +`python.buildEnv` should be used with `ignoreCollisions = true`. + +#### Setup hooks {#setup-hooks} + +The following are setup hooks specifically for Python packages. Most of these +are used in `buildPythonPackage`. + +- `eggUnpackhook` to move an egg to the correct folder so it can be installed + with the `eggInstallHook` +- `eggBuildHook` to skip building for eggs. +- `eggInstallHook` to install eggs. +- `flitBuildHook` to build a wheel using `flit`. +- `pipBuildHook` to build a wheel using `pip` and PEP 517. Note a build system + (e.g. `setuptools` or `flit`) should still be added as `nativeBuildInput`. +- `pipInstallHook` to install wheels. +- `pytestCheckHook` to run tests with `pytest`. See [example usage](#using-pytestcheckhook). +- `pythonCatchConflictsHook` to check whether a Python package is not already existing. +- `pythonImportsCheckHook` to check whether importing the listed modules works. +- `pythonRemoveBinBytecode` to remove bytecode from the `/bin` folder. +- `setuptoolsBuildHook` to build a wheel using `setuptools`. +- `setuptoolsCheckHook` to run tests with `python setup.py test`. +- `venvShellHook` to source a Python 3 `venv` at the `venvDir` location. A + `venv` is created if it does not yet exist. `postVenvCreation` can be used to + to run commands only after venv is first created. +- `wheelUnpackHook` to move a wheel to the correct folder so it can be installed + with the `pipInstallHook`. + +### Development mode {#development-mode} + +Development or editable mode is supported. To develop Python packages +`buildPythonPackage` has additional logic inside `shellPhase` to run `pip +install -e . --prefix $TMPDIR/`for the package. + +Warning: `shellPhase` is executed only if `setup.py` exists. + +Given a `default.nix`: + +```nix +with import <nixpkgs> {}; + +pythonPackages.buildPythonPackage { + name = "myproject"; + buildInputs = with pythonPackages; [ pyramid ]; + + src = ./.; +} +``` + +Running `nix-shell` with no arguments should give you the environment in which +the package would be built with `nix-build`. + +Shortcut to setup environments with C headers/libraries and Python packages: + +```shell +nix-shell -p pythonPackages.pyramid zlib libjpeg git +``` + +Note: There is a boolean value `lib.inNixShell` set to `true` if nix-shell is invoked. + +### Tools {#tools} + +Packages inside nixpkgs are written by hand. However many tools exist in +community to help save time. No tool is preferred at the moment. + +- [pypi2nix](https://github.com/nix-community/pypi2nix): Generate Nix + expressions for your Python project. Note that [sharing derivations from + pypi2nix with nixpkgs is possible but not + encouraged](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376). +- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools) +- [poetry2nix](https://github.com/nix-community/poetry2nix) + +### Deterministic builds {#deterministic-builds} + +The Python interpreters are now built deterministically. Minor modifications had +to be made to the interpreters in order to generate deterministic bytecode. This +has security implications and is relevant for those using Python in a +`nix-shell`. + +When the environment variable `DETERMINISTIC_BUILD` is set, all bytecode will +have timestamp 1. The `buildPythonPackage` function sets `DETERMINISTIC_BUILD=1` +and [PYTHONHASHSEED=0](https://docs.python.org/3.8/using/cmdline.html#envvar-PYTHONHASHSEED). +Both are also exported in `nix-shell`. + +### Automatic tests {#automatic-tests} + +It is recommended to test packages as part of the build process. +Source distributions (`sdist`) often include test files, but not always. + +By default the command `python setup.py test` is run as part of the +`checkPhase`, but often it is necessary to pass a custom `checkPhase`. An +example of such a situation is when `py.test` is used. + +#### Common issues {#common-issues} + +* Non-working tests can often be deselected. By default `buildPythonPackage` + runs `python setup.py test`. Most Python modules follows the standard test + protocol where the pytest runner can be used instead. `py.test` supports a + `-k` parameter to ignore test methods or classes: + + ```nix + buildPythonPackage { + # ... + # assumes the tests are located in tests + checkInputs = [ pytest ]; + checkPhase = '' + py.test -k 'not function_name and not other_function' tests + ''; + } + ``` + +* Tests that attempt to access `$HOME` can be fixed by using the following + work-around before running tests (e.g. `preCheck`): `export HOME=$(mktemp -d)` + +## FAQ {#faq} + +### How to solve circular dependencies? {#how-to-solve-circular-dependencies} + +Consider the packages `A` and `B` that depend on each other. When packaging `B`, +a solution is to override package `A` not to depend on `B` as an input. The same +should also be done when packaging `A`. + +### How to override a Python package? {#how-to-override-a-python-package} + +We can override the interpreter and pass `packageOverrides`. In the following +example we rename the `pandas` package and build it. + +```nix +with import <nixpkgs> {}; + +(let + python = let + packageOverrides = self: super: { + pandas = super.pandas.overridePythonAttrs(old: {name="foo";}); + }; + in pkgs.python38.override {inherit packageOverrides;}; + +in python.withPackages(ps: [ps.pandas])).env +``` + +Using `nix-build` on this expression will build an environment that contains the +package `pandas` but with the new name `foo`. + +All packages in the package set will use the renamed package. A typical use case +is to switch to another version of a certain package. For example, in the +Nixpkgs repository we have multiple versions of `django` and `scipy`. In the +following example we use a different version of `scipy` and create an +environment that uses it. All packages in the Python package set will now use +the updated `scipy` version. + +```nix +with import <nixpkgs> {}; + +( let + packageOverrides = self: super: { + scipy = super.scipy_0_17; + }; + in (pkgs.python38.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze]) +).env +``` + +The requested package `blaze` depends on `pandas` which itself depends on `scipy`. + +If you want the whole of Nixpkgs to use your modifications, then you can use +`overlays` as explained in this manual. In the following example we build a +`inkscape` using a different version of `numpy`. + +```nix +let + pkgs = import <nixpkgs> {}; + newpkgs = import pkgs.path { overlays = [ (self: super: { + python38 = let + packageOverrides = python-self: python-super: { + numpy = python-super.numpy_1_18; + }; + in super.python38.override {inherit packageOverrides;}; + } ) ]; }; +in newpkgs.inkscape +``` + +### `python setup.py bdist_wheel` cannot create .whl {#python-setup.py-bdist_wheel-cannot-create-.whl} + +Executing `python setup.py bdist_wheel` in a `nix-shell `fails with +``` +ValueError: ZIP does not support timestamps before 1980 +``` + +This is because files from the Nix store (which have a timestamp of the UNIX +epoch of January 1, 1970) are included in the .ZIP, but .ZIP archives follow the +DOS convention of counting timestamps from 1980. + +The command `bdist_wheel` reads the `SOURCE_DATE_EPOCH` environment variable, +which `nix-shell` sets to 1. Unsetting this variable or giving it a value +corresponding to 1980 or later enables building wheels. + +Use 1980 as timestamp: + +```shell +nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel" +``` + +or the current time: + +```shell +nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel" +``` + +or unset `SOURCE_DATE_EPOCH`: + +```shell +nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel" +``` + +### `install_data` / `data_files` problems {#install_data-data_files-problems} + +If you get the following error: + +``` +could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc': +Permission denied +``` + +This is a [known bug](https://github.com/pypa/setuptools/issues/130) in +`setuptools`. Setuptools `install_data` does not respect `--prefix`. An example +of such package using the feature is `pkgs/tools/X11/xpra/default.nix`. + +As workaround install it as an extra `preInstall` step: + +```shell +${python.interpreter} setup.py install_data --install-dir=$out --root=$out +sed -i '/ = data\_files/d' setup.py +``` + +### Rationale of non-existent global site-packages {#rationale-of-non-existent-global-site-packages} + +On most operating systems a global `site-packages` is maintained. This however +becomes problematic if you want to run multiple Python versions or have multiple +versions of certain libraries for your projects. Generally, you would solve such +issues by creating virtual environments using `virtualenv`. + +On Nix each package has an isolated dependency tree which, in the case of +Python, guarantees the right versions of the interpreter and libraries or +packages are available. There is therefore no need to maintain a global `site-packages`. + +If you want to create a Python environment for development, then the recommended +method is to use `nix-shell`, either with or without the `python.buildEnv` +function. + +### How to consume Python modules using pip in a virtual environment like I am used to on other Operating Systems? {#how-to-consume-python-modules-using-pip-in-a-virtual-environment-like-i-am-used-to-on-other-operating-systems} + +While this approach is not very idiomatic from Nix perspective, it can still be +useful when dealing with pre-existing projects or in situations where it's not +feasible or desired to write derivations for all required dependencies. + +This is an example of a `default.nix` for a `nix-shell`, which allows to consume +a virtual environment created by `venv`, and install Python modules through +`pip` the traditional way. + +Create this `default.nix` file, together with a `requirements.txt` and simply +execute `nix-shell`. + +```nix +with import <nixpkgs> { }; + +let + pythonPackages = python3Packages; +in pkgs.mkShell rec { + name = "impurePythonEnv"; + venvDir = "./.venv"; + buildInputs = [ + # A Python interpreter including the 'venv' module is required to bootstrap + # the environment. + pythonPackages.python + + # This execute some shell code to initialize a venv in $venvDir before + # dropping into the shell + pythonPackages.venvShellHook + + # Those are dependencies that we would like to use from nixpkgs, which will + # add them to PYTHONPATH and thus make them accessible from within the venv. + pythonPackages.numpy + pythonPackages.requests + + # In this particular example, in order to compile any binary extensions they may + # require, the Python modules listed in the hypothetical requirements.txt need + # the following packages to be installed locally: + taglib + openssl + git + libxml2 + libxslt + libzip + zlib + ]; + + # Run this command, only after creating the virtual environment + postVenvCreation = '' + unset SOURCE_DATE_EPOCH + pip install -r requirements.txt + ''; + + # Now we can execute any commands within the virtual environment. + # This is optional and can be left out to run pip manually. + postShellHook = '' + # allow pip to install wheels + unset SOURCE_DATE_EPOCH + ''; + +} +``` + +In case the supplied venvShellHook is insufficient, or when Python 2 support is +needed, you can define your own shell hook and adapt to your needs like in the +following example: + +```nix +with import <nixpkgs> { }; + +let + venvDir = "./.venv"; + pythonPackages = python3Packages; +in pkgs.mkShell rec { + name = "impurePythonEnv"; + buildInputs = [ + pythonPackages.python + # Needed when using python 2.7 + # pythonPackages.virtualenv + # ... + ]; + + # This is very close to how venvShellHook is implemented, but + # adapted to use 'virtualenv' + shellHook = '' + SOURCE_DATE_EPOCH=$(date +%s) + + if [ -d "${venvDir}" ]; then + echo "Skipping venv creation, '${venvDir}' already exists" + else + echo "Creating new venv environment in path: '${venvDir}'" + # Note that the module venv was only introduced in python 3, so for 2.7 + # this needs to be replaced with a call to virtualenv + ${pythonPackages.python.interpreter} -m venv "${venvDir}" + fi + + # Under some circumstances it might be necessary to add your virtual + # environment to PYTHONPATH, which you can do here too; + # PYTHONPATH=$PWD/${venvDir}/${pythonPackages.python.sitePackages}/:$PYTHONPATH + + source "${venvDir}/bin/activate" + + # As in the previous example, this is optional. + pip install -r requirements.txt + ''; +} +``` + +Note that the `pip install` is an imperative action. So every time `nix-shell` +is executed it will attempt to download the Python modules listed in +requirements.txt. However these will be cached locally within the `virtualenv` +folder and not downloaded again. + +### How to override a Python package from `configuration.nix`? {#how-to-override-a-python-package-from-configuration.nix} + +If you need to change a package's attribute(s) from `configuration.nix` you could do: + +```nix + nixpkgs.config.packageOverrides = super: { + python = super.python.override { + packageOverrides = python-self: python-super: { + twisted = python-super.twisted.overrideAttrs (oldAttrs: { + src = super.fetchPypi { + pname = "twisted"; + version = "19.10.0"; + sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d"; + extension = "tar.bz2"; + }; + }); + }; + }; + }; +``` + +`pythonPackages.twisted` is now globally overridden. +All packages and also all NixOS services that reference `twisted` +(such as `services.buildbot-worker`) now use the new definition. +Note that `python-super` refers to the old package set and `python-self` +to the new, overridden version. + +To modify only a Python package set instead of a whole Python derivation, use +this snippet: + +```nix + myPythonPackages = pythonPackages.override { + overrides = self: super: { + twisted = ...; + }; + } +``` + +### How to override a Python package using overlays? {#how-to-override-a-python-package-using-overlays} + +Use the following overlay template: + +```nix +self: super: { + python = super.python.override { + packageOverrides = python-self: python-super: { + twisted = python-super.twisted.overrideAttrs (oldAttrs: { + src = super.fetchPypi { + pname = "twisted"; + version = "19.10.0"; + sha256 = "7394ba7f272ae722a74f3d969dcf599bc4ef093bc392038748a490f1724a515d"; + extension = "tar.bz2"; + }; + }); + }; + }; +} +``` + +### How to use Intel’s MKL with numpy and scipy? {#how-to-use-intels-mkl-with-numpy-and-scipy} + +MKL can be configured using an overlay. See the section "[Using overlays to +configure alternatives](#sec-overlays-alternatives-blas-lapack)". + +### What inputs do `setup_requires`, `install_requires` and `tests_require` map to? {#what-inputs-do-setup_requires-install_requires-and-tests_require-map-to} + +In a `setup.py` or `setup.cfg` it is common to declare dependencies: + +* `setup_requires` corresponds to `nativeBuildInputs` +* `install_requires` corresponds to `propagatedBuildInputs` +* `tests_require` corresponds to `checkInputs` + +## Contributing {#contributing} + +### Contributing guidelines {#contributing-guidelines} + +The following rules are desired to be respected: + +* Python libraries are called from `python-packages.nix` and packaged with + `buildPythonPackage`. The expression of a library should be in + `pkgs/development/python-modules/<name>/default.nix`. +* Python applications live outside of `python-packages.nix` and are packaged + with `buildPythonApplication`. +* Make sure libraries build for all Python interpreters. +* By default we enable tests. Make sure the tests are found and, in the case of + libraries, are passing for all interpreters. If certain tests fail they can be + disabled individually. Try to avoid disabling the tests altogether. In any + case, when you disable tests, leave a comment explaining why. +* Commit names of Python libraries should reflect that they are Python + libraries, so write for example `pythonPackages.numpy: 1.11 -> 1.12`. +* Attribute names in `python-packages.nix` as well as `pname`s should match the + library's name on PyPI, but be normalized according to [PEP + 0503](https://www.python.org/dev/peps/pep-0503/#normalized-names). This means + that characters should be converted to lowercase and `.` and `_` should be + replaced by a single `-` (foo-bar-baz instead of Foo__Bar.baz). + If necessary, `pname` has to be given a different value within `fetchPypi`. +* Attribute names in `python-packages.nix` should be sorted alphanumerically to + avoid merge conflicts and ease locating attributes. + +## Package set maintenance + +The whole Python package set has a lot of packages that do not see regular +updates, because they either are a very fragile component in the Python +ecosystem, like for example the `hypothesis` package, or packages that have +no maintainer, so maintenance falls back to the package set maintainers. + +### Updating packages in bulk + +There is a tool to update alot of python libraries in bulk, it exists at +`maintainers/scripts/update-python-libraries` with this repository. + +It can quickly update minor or major versions for all packages selected +and create update commits, and supports the `fetchPypi`, `fetchurl` and +`fetchFromGitHub` fetchers. When updating lots of packages that are +hosted on GitHub, exporting a `GITHUB_API_TOKEN` is highly recommended. + +Updating packages in bulk leads to lots of breakages, which is why a +stabilization period on the `python-unstable` branch is required. + +Once the branch is sufficiently stable it should normally be merged +into the `staging` branch. + +An exemplary call to update all python libraries between minor versions +would be: + +```ShellSession +$ maintainers/scripts/update-python-libraries --target minor --commit --use-pkgs-prefix pkgs/development/python-modules/**/default.nix +``` + +## CPython Update Schedule + +With [PEP 602](https://www.python.org/dev/peps/pep-0602/), CPython now +follows a yearly release cadence. In nixpkgs, all supported interpreters +are made available, but only the most recent two +interpreters package sets are built; this is a compromise between being +the latest interpreter, and what the majority of the Python packages support. + +New CPython interpreters are released in October. Generally, it takes some +time for the majority of active Python projects to support the latest stable +interpreter. To help ease the migration for Nixpkgs users +between Python interpreters the schedule below will be used: + +| When | Event | +| --- | --- | +| After YY.11 Release | Bump CPython package set window. The latest and previous latest stable should now be built. | +| After YY.05 Release | Bump default CPython interpreter to latest stable. | + +In practice, this means that the Python community will have had a stable interpreter +for ~2 months before attempting to update the package set. And this will +allow for ~7 months for Python applications to support the latest interpreter. |