Packaging Your First Experiment¶

In previous sections, we have seen how to define a Nix package and how to use nix-shell to enter a controlled environment or to run commands from within such an environment. In this section we will set up a first repeatable experiment based on a Nix environment.

The toy experiment that we will do here consists in running one instance of the Chord simulator discussed earlier. This experiment is defined in the first-experiment directory of the Chord experiments git repository. The following commands retrieve the files then places you into the proper directory.

git clone https://gitlab.inria.fr/nix-tutorial/chord-experiments.git /tmp/chord-experiments

cd /tmp/chord-experiments/first-experiment

Experiment structure¶

This directory contains the following input files, necessary to replay the experiment:

A readme (README.md) that describes the experiment and how to run it.
A SimGrid platform file cluster_backbone.xml.
A SimGrid deployment file s4u-dht-chord_d.xml.
Runner scripts runner.sh and runner_shebang.sh that run the experiment.
A default.nix Nix file that contains the chord package and a shell used to run the experiment.

default.nix contains the following.

first experiment's default.nix¶

{
  pkgs ? import (fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/4fe8d07066f6ea82cda2b0c9ae7aee59b2d241b3.tar.gz";
    sha256 = "sha256:06jzngg5jm1f81sc4xfskvvgjy5bblz51xpl788mnps1wrkykfhp";
  }) {}
}:

with pkgs;

let
  packages = rec {

    # The derivation for chord
    chord = stdenv.mkDerivation rec {
      pname = "chord";
      version = "0.0.in-tuto";
      src = fetchgit {
        url = "https://gitlab.inria.fr/nix-tutorial/chord-tuto-nix-2022";
        rev = "069d2a5bfa4c4024063c25551d5201aeaf921cb3";
        sha256 = "sha256-MlqJOoMSRuYeG+jl8DFgcNnpEyeRgDCK2JlN9pOqBWA=";
      };

      buildInputs = [
        pkgconfig
        simgrid
        boost
        cmake
      ];

    };

    # The shell of our experiment runtime environment
    expEnv = mkShell rec {
      name = "exp01Env";
      buildInputs = [
        chord
      ];
    };

  };
in
  packages

This file looks similar to the one discussed in previous section, but its structure changed a little. The main difference is that the file does not return a single derivation (the chord package) but a set with several attributes.

A chord package, very similar to the one presented in previous section.
A expEnv shell environment, meant to be used to run the experiment.

The advantage of this structure over the previous one is that we can easily define many packages and environment with a single Nix file.

Note

Here, the expEnv attribute can refer to the chord attribute that is defined within the same set. This is possible thanks to the packages recursive set — a non-recursive set would not allow this.

Both nix-shell and nix-build can be used with this file. However, we should tell the commands on which attribute they should work within the set, thanks to the --attr (-A) command-line option. For instance, the chord package can be built with the following command.

nix-build default.nix -A chord

Run the experiment manually¶

The command to enter into the runtime environment of this experiment is the following.

nix-shell default.nix -A expEnv

From within the runtime environment, the chord simulator should be in your path.

chord --version
# should NOT trigger a 'command not found' error

The given runner.sh runs the chord executable on specified inputs.

first experiment's runner.sh¶

#!/usr/bin/env bash
chord cluster_backbone.xml s4u-dht-chord_d.xml

From within the runtime environment, running the experiment is straightforward.

./runner.sh

The experiment result should exactly be:

[node-1.simgrid.org:node:(2) 10.000000] [s4u_chord/INFO] Joining the ring with id 366680, knowing node 42
[node-2.simgrid.org:node:(3) 20.000000] [s4u_chord/INFO] Joining the ring with id 533744, knowing node 366680
[node-3.simgrid.org:node:(4) 30.000000] [s4u_chord/INFO] Joining the ring with id 1319738, knowing node 42
[node-4.simgrid.org:node:(5) 40.000000] [s4u_chord/INFO] Joining the ring with id 16509405, knowing node 366680
[node-9.simgrid.org:node:(10) 60.000000] [s4u_chord/INFO] Joining the ring with id 16725096, knowing node 1319738
[node-8.simgrid.org:node:(9) 60.000000] [s4u_chord/INFO] Joining the ring with id 16728496, knowing node 1319738
[node-7.simgrid.org:node:(8) 60.000000] [s4u_chord/INFO] Joining the ring with id 16728094, knowing node 1319738
[node-6.simgrid.org:node:(7) 60.000000] [s4u_chord/INFO] Joining the ring with id 16728090, knowing node 1319738
[node-3.simgrid.org:node:(4) 235.021718] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-1.simgrid.org:node:(2) 235.021718] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-5.simgrid.org:node:(6) 250.000000] [s4u_chord/INFO] Joining the ring with id 10874876, knowing node 533744
[node-4.simgrid.org:node:(5) 360.035830] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-2.simgrid.org:node:(3) 440.035230] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-5.simgrid.org:node:(6) 850.094479] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-9.simgrid.org:node:(10) 860.094279] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-6.simgrid.org:node:(7) 865.071761] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-7.simgrid.org:node:(8) 895.094579] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-8.simgrid.org:node:(9) 920.071661] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[node-0.simgrid.org:node:(1) 1045.063355] [s4u_chord/INFO] Well Guys! I Think it's time for me to leave ;)
[1145.063355] [s4u_chord/INFO] Simulated time: 1145.06

Alternative launching options¶

Manually running commands from within a nix-shell environment is convenient for routine tasks, but this is not great to automate the experiment launch. Here are alternatives that make it possible.

nix-shell’s `--command`¶

As seen in previous sections, nix-shell’s --command option is very convenient for such operations.

nix-shell default.nix -A expEnv --command ./runner.sh

`mkShell`’s `shellHook`¶

Another interesting feature is the ability to specify the command to execute directly within the mkShell function. This is done by specifying a shellHook attribute within the set given to mkShell.

expEnvWithHook = mkShell rec {
  name = "exp01Env";
  buildInputs = [
    chord
  ];
  shellHook = "./runner.sh";
};

nix-shell shebang¶

Finally, the runtime environment of a script can be defined with a nix-shell shebang (see Nix’s documentation on nix-shell shebangs).

first experiment's runner_shebang.sh¶

#!/usr/bin/env nix-shell
#!nix-shell default.nix -A expEnv -i bash
chord cluster_backbone.xml s4u-dht-chord_d.xml

With this solution, manually calling nix-shell is not required, as the runner will do it for us. Simply launch the runner as any shell script, and it will automatically load the environment and run the script.

./runner_shebang.sh

Warning

The Nix expression the shebang refers to (default.nix in this example) must be relative to the script.

This is less powerful than a generic nix-shell call, which can use expressions defined in remote repositories. For example, the following command enters into the build environment of Batsim’s last stable release, as defined in the kapack package repository.

nix-shell https://github.com/oar-team/nur-kapack/archive/master.tar.gz -A batsim

Building a Docker Image¶

It can often be interesting to provide a Docker image of an application for portability reasons. However, a Dockerfile is also victim of some reproducibility flaws. Indeed, calling apt get update already makes the building of the image not reproducible as it depends on the state of the package repository at the moment of the build. Besides, the traceability of the build is often blurry and difficult to do properly.

Nix can build reproducible Docker images from a Nix derivation.

Let us keep the example of the chord application. We can extend the previous default.nix with a new derivation called chord-docker:

{
  pkgs ? import (fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/4fe8d07066f6ea82cda2b0c9ae7aee59b2d241b3.tar.gz";
    sha256 = "sha256:06jzngg5jm1f81sc4xfskvvgjy5bblz51xpl788mnps1wrkykfhp";
  }) {}
}:

with pkgs;

let
  packages = rec {

    # The derivation for chord
    chord = stdenv.mkDerivation rec {
      pname = "chord";
      version = "0.0.in-tuto";
      src = fetchgit {
        url = "https://gitlab.inria.fr/nix-tutorial/chord-tuto-nix-2022";
        rev = "069d2a5bfa4c4024063c25551d5201aeaf921cb3";
        sha256 = "sha256-MlqJOoMSRuYeG+jl8DFgcNnpEyeRgDCK2JlN9pOqBWA=";
      };

      buildInputs = [
        pkgconfig
        simgrid
        boost
        cmake
      ];

    };

    chord-docker = dockerTools.buildImage {
      name = "chord-docker";
      tag = "tuto-nix";
      contents = [ chord ];
      config = {
        Cmd = [ "${chord}/bin/chord" ];
        WorkingDir = "/data";
        Volumes = { "/data" = { }; };
      };
    };

    # The shell of our experiment runtime environment
    expEnv = mkShell rec {
      name = "exp01Env";
      buildInputs = [
        chord
      ];
    };

  };
in
  packages

The contents field of dockerTools.buildImage are the runtime dependencies to include in the container. In our case, we only need chord which does not need any of its own. The config set is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the Docker Image Specification. More examples of Docker images built with Nix are available here and the Nix documentation on Docker there.

To build the Docker image, we first build the derivation:

nix-build default.nix -A chord-docker

It will generates a tarball in result that we need to load with docker.

docker load < result

The container is now loaded and can be ran as usual:

docker run -v $(pwd):/data chord-docker:tuto-nix chord /data/cluster_backbone.xml /data/s4u-dht-chord_d.xml

Packaging Your First Experiment¶

Experiment structure¶

Run the experiment manually¶

Alternative launching options¶

nix-shell’s --command¶

mkShell’s shellHook¶

nix-shell shebang¶

Building a Docker Image¶

nix-shell’s `--command`¶

`mkShell`’s `shellHook`¶