Packaging Your First Experiment =============================== In previous sections, we have seen how to define a Nix package and how to use ``nix-shell`` to enter a controlled environment or to run commands from within such an environment. In this section we will set up a first repeatable experiment based on a Nix environment. The toy experiment that we will do here consists in running one instance of the Chord simulator discussed earlier. This experiment is defined in the ``first-experiment`` directory of the `Chord experiments git repository`_. The following commands retrieve the files then places you into the proper directory. .. literalinclude:: ./first-experiment/retrieve-repo.bash :lines: 3 :language: bash .. literalinclude:: ./first-experiment/move-into-repo.bash :lines: 2 :language: bash Experiment structure -------------------- This directory contains the following input files, necessary to replay the experiment: - A readme (``README.md``) that describes the experiment and how to run it. - A SimGrid platform file ``cluster_backbone.xml``. - A SimGrid deployment file ``s4u-dht-chord_d.xml``. - Runner scripts ``runner.sh`` and ``runner_shebang.sh`` that run the experiment. - A ``default.nix`` Nix file that contains the ``chord`` package and a shell used to run the experiment. ``default.nix`` contains the following. .. literalinclude:: ./examples/chord-experiments/first-experiment/default.nix :caption: :download:`first experiment's default.nix <./examples/chord-experiments/first-experiment/default.nix>` :name: first-experiment-default.nix :language: nix This file looks similar to the one discussed in previous section, but its structure changed a little. The main difference is that the file does not return a single *derivation* (the ``chord`` package) but a *set* with several *attributes*. - A ``chord`` package, very similar to the one presented in previous section. - A ``expEnv`` shell environment, meant to be used to run the experiment. The advantage of this structure over the previous one is that we can easily define many packages and environment with a single Nix file. .. note:: Here, the ``expEnv`` attribute can refer to the ``chord`` attribute that is defined within the same set. This is possible thanks to the ``packages`` `recursive set`_ — a non-recursive set would not allow this. Both ``nix-shell`` and ``nix-build`` can be used with this file. However, we should tell the commands on which attribute they should work within the set, thanks to the ``--attr`` (``-A``) command-line option. For instance, the chord package can be built with the following command. .. code-block:: bash nix-build default.nix -A chord Run the experiment manually --------------------------- The command to enter into the runtime environment of this experiment is the following. .. code-block:: bash nix-shell default.nix -A expEnv From within the runtime environment, the chord simulator should be in your path. .. code-block:: bash chord --version # should NOT trigger a 'command not found' error The given ``runner.sh`` runs the `chord` executable on specified inputs. .. literalinclude:: ./examples/chord-experiments/first-experiment/runner.sh :caption: :download:`first experiment's runner.sh <./examples/chord-experiments/first-experiment/runner.sh>` :language: bash From within the runtime environment, running the experiment is straightforward. .. code-block:: bash ./runner.sh The experiment result should **exactly** be: .. literalinclude:: ./first-experiment/expected-result.out Alternative launching options ----------------------------- Manually running commands from within a nix-shell environment is convenient for routine tasks, but this is not great to automate the experiment launch. Here are alternatives that make it possible. nix-shell's ``--command`` +++++++++++++++++++++++++ As seen in previous sections, nix-shell's ``--command`` option is very convenient for such operations. .. literalinclude:: ./first-experiment/run-nixshell-cmd.bash :lines: 2 :language: bash ``mkShell``'s ``shellHook`` +++++++++++++++++++++++++++ Another interesting feature is the ability to specify the command to execute directly within the ``mkShell`` function. This is done by specifying a ``shellHook`` attribute within the set given to ``mkShell``. .. code-block:: nix expEnvWithHook = mkShell rec { name = "exp01Env"; buildInputs = [ chord ]; shellHook = "./runner.sh"; }; nix-shell shebang +++++++++++++++++ Finally, the runtime environment of a script can be defined with a nix-shell shebang (see `Nix's documentation on nix-shell shebangs`_). .. literalinclude:: ./examples/chord-experiments/first-experiment/runner_shebang.sh :caption: :download:`first experiment's runner_shebang.sh <./examples/chord-experiments/first-experiment/runner_shebang.sh>` :language: bash With this solution, manually calling ``nix-shell`` is not required, as the runner will do it for us. Simply launch the runner as any shell script, and it will automatically load the environment and run the script. .. literalinclude:: ./first-experiment/run-nixshell-shebang.bash :lines: 2 :language: bash .. warning:: The Nix expression the shebang refers to (``default.nix`` in this example) must be relative to the script. This is less powerful than a generic ``nix-shell`` call, which can use expressions defined in remote repositories. For example, the following command enters into the build environment of Batsim_'s last stable release, as defined in the kapack_ package repository. .. code-block:: bash nix-shell https://github.com/oar-team/nur-kapack/archive/master.tar.gz -A batsim Building a Docker Image ----------------------- It can often be interesting to provide a Docker image of an application for portability reasons. However, a ``Dockerfile`` is also victim of some reproducibility flaws. Indeed, calling `apt get update` already makes the building of the image not reproducible as it depends on the state of the package repository at the moment of the build. Besides, the traceability of the build is often blurry and difficult to do properly. Nix can build **reproducible Docker images** from a Nix derivation. Let us keep the example of the ``chord`` application. We can extend the previous ``default.nix`` with a new derivation called ``chord-docker``: .. literalinclude:: ./examples/chord-experiments/first-experiment/default-with-docker.nix :language: nix The ``contents`` field of ``dockerTools.buildImage`` are the runtime dependencies to include in the container. In our case, we only need ``chord`` which does not need any of its own. The ``config`` set is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the `Docker Image Specification`_. More examples of Docker images built with Nix are available `here `_ and the Nix documentation on Docker `there `_. To build the Docker image, we first build the derivation: .. code-block:: bash nix-build default.nix -A chord-docker It will generates a tarball in ``result`` that we need to load with docker. .. code-block:: bash docker load < result The container is now loaded and can be ran as usual: .. code-block:: bash docker run -v $(pwd):/data chord-docker:tuto-nix chord /data/cluster_backbone.xml /data/s4u-dht-chord_d.xml .. _Chord experiments git repository: https://gitlab.inria.fr/nix-tutorial/chord-experiments .. _recursive set: https://nixos.wiki/wiki/Nix_Expression_Language#rec_statement .. _Batsim: http://batsim.rtfd.io/ .. _kapack: https://github.com/oar-team/nur-kapack/ .. _Nix's documentation on nix-shell shebangs: https://nixos.org/nix/manual/#use-as-a-interpreter .. _Building and running Docker images: https://nixos.org/guides/building-and-running-docker-images.html .. _More examples of Docker images with Nix: https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/docker/examples.nix .. _Docker Image Specification: https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions