.. _section_expe_packaging_do_not_repeat_yourself: Experiment Packaging: Don't Repeat Yourself =========================================== The previous section showed how to create a reproducible experiment with Nix. The structure of the experiment was a single ``default.nix`` file that directly describes the various packages and environments needed for the experiment. While this structure is fine to manage a single experiment, it is far from optimal when you conduct a series of experiment with intersecting software — e.g., if you are doing a PhD. .. image:: http://phdcomics.com/comics/archive/phd092316s.gif A very simple solution is to define a file like the :ref:`first-experiment-default.nix` in each experiment, but it would create code replication and be hard to maintain. In this section, we will see how to define a central repository that contains many packages needed by the experiments, and how to use it in an experiment. Starting your own (tiny) package repository ------------------------------------------- Nix enables decentralized package definitions, as the packages do not need to be in the same repository to be well defined. Here we will present a convenient structure to define our own package repository. .. note:: An example of this repository is available `here `_. It can be used as a template for future experiments. This structure is compatible with `Nix User Repository`_ recommendations. The main file of the repository is the ``default.nix`` located at the repository root. It is very important, as it is the entry point of the repository. This file should define all the packages that are to be exported — but this does not mean that all the package definitions should be in this file. .. literalinclude:: ./examples/packages-repository/default.nix :caption: :download:`package repository's /default.nix <./examples/packages-repository/default.nix>` :language: nix :name: packages-repository-default.nix Similar to the :ref:`first-experiment-default.nix`, this file returns a *set* of packages. The most significant change is the introduction to the ``callPackage`` function, which simplifies the way to call a derivation. More information about ``callPackage`` can be found in the `Nix pill about callPackage`_. The derivation for the *chord* simulator has been moved into the ``pkgs/chord/default.nix`` file. Creating a directory for each package is recommended, as it allows to store various versions of the package in the same directory. ``callPackage`` takes two arguments: A path to a Nix file (or to a directory into which it will look for a ``default.nix`` file), and a *set* used to override the inputs of the called package. Here is the content of the chord package. .. literalinclude:: ./examples/packages-repository/pkgs/chord/default.nix :caption: :download:`pkgs/chord/default.nix <./examples/packages-repository/pkgs/chord/default.nix>` :language: nix Instead of taking the whole ``pkgs`` as input as in :ref:`first-experiment-default.nix`, this file finely specifies its inputs. This is done on the first line, which here defines these inputs: `stdenv`, `fetchurl`, `cmake`, `simgrid` and `boost`. Please note that for ``pkgs/chord/default.nix`` taken in isolation, these inputs are not defined as they do not have a default value. This file is meant to be used with ``callPackage``, whose role is to call the package by giving it the requested inputs. This structure makes it easy to define variations of a package. This is for example done in the ``chord_custom_sg`` attribute of :ref:`packages-repository-default.nix`, which uses a custom version of SimGrid instead of the one defined in Nixpkgs. The custom SimGrid version here is very similar to the one defined in Nixpkgs, as we just wanted to change the SimGrid commit to use. For more information on package overriding, please refer to the `Nix pill on overriding packages`_ and to the `Nix documentation on overriding`_. .. literalinclude:: ./examples/packages-repository/pkgs/simgrid/custom.nix :caption: :download:`pkgs/simgrid/custom.nix <./examples/packages-repository/pkgs/simgrid/custom.nix>` :language: nix Usage ^^^^^ Now that the package repository is set up, it can be used directly from ``nix-build`` or ``nix-shell``. For example, the following commands builds the ``chord`` package defined in the package repository. .. code-block:: bash nix-build https://gitlab.inria.fr/nix-tutorial/packages-repository/-/archive/master/packages-repository-master.tar.gz -A chord Alternatively, the package repository can of course be fetched locally manually first. .. code-block:: bash git clone https://gitlab.inria.fr/nix-tutorial/packages-repository.git /tmp/packages-repository nix-build /tmp/packages-repository -A chord Package an experiment using the tiny package repository ------------------------------------------------------- Defining an experiment that uses the tiny package repository we have just defined is very similar to what we did in the previous experiment. This is done in the ``second-experiment`` directory of the `Chord experiments git repository`_. Here are commands to retrieve a copy of the experiment and to move into the experiment directory. .. literalinclude:: ./second-experiment/retrieve-repo.bash :lines: 3 :language: bash .. literalinclude:: ./second-experiment/move-into-repo.bash :lines: 2 :language: bash Most files are very similar to those of the previous experiment. The main difference is that we chose here to provide a ``shell.nix`` file instead of a ``default.nix`` file. This is because in our special case we only define a single environment, the one to run the experiment. Here, ``shell.nix`` simply imports our tiny package repository and defines a shell that uses the ``chord`` package defined in the tiny package repository. .. literalinclude:: ./examples/chord-experiments/second-experiment/shell.nix :caption: :download:`second experiment's shell.nix <./examples/chord-experiments/second-experiment/shell.nix>` :name: second-experiment-shell.nix :language: nix The shebang runner have been slightly adapted to use ``shell.nix``. .. literalinclude:: ./examples/chord-experiments/second-experiment/runner_shebang.sh :caption: :download:`second experiment's runner_shebang.sh <./examples/chord-experiments/second-experiment/runner_shebang.sh>` :language: bash .. _Chord experiments git repository: https://gitlab.inria.fr/nix-tutorial/chord-experiments .. _Nix pill about callPackage: https://nixos.org/nixos/nix-pills/callpackage-design-pattern.html .. _Nix User Repository: https://github.com/nix-community/NUR .. _Nix pill on overriding packages: https://nixos.org/nixos/nix-pills/override-design-pattern.html .. _Nix documentation on overriding: https://nixos.org/nixpkgs/manual/#sec-overrides