Experiment Packaging: Don’t Repeat Yourself¶
The previous section showed how to create a reproducible experiment with Nix.
The structure of the experiment was a single default.nix file that directly describes the various packages and environments needed for the experiment.
While this structure is fine to manage a single experiment, it is far from optimal when you conduct a series of experiment with intersecting software — e.g., if you are doing a PhD.
A very simple solution is to define a file like the first experiment's default.nix in each experiment, but it would create code replication and be hard to maintain. In this section, we will see how to define a central repository that contains many packages needed by the experiments, and how to use it in an experiment.
Starting your own (tiny) package repository¶
Nix enables decentralized package definitions, as the packages do not need to be in the same repository to be well defined. Here we will present a convenient structure to define our own package repository.
Note
An example of this repository is available here. It can be used as a template for future experiments. This structure is compatible with Nix User Repository recommendations.
The main file of the repository is the default.nix located at the repository root.
It is very important, as it is the entry point of the repository.
This file should define all the packages that are to be exported — but this does not mean that all the package definitions should be in this file.
{
pkgs ? import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4fe8d07066f6ea82cda2b0c9ae7aee59b2d241b3.tar.gz";
sha256 = "sha256:06jzngg5jm1f81sc4xfskvvgjy5bblz51xpl788mnps1wrkykfhp";
}) {}
}:
with pkgs;
let
packages = rec {
chord = callPackage ./pkgs/chord {};
chord_custom_sg = callPackage ./pkgs/chord { simgrid = custom_simgrid; };
custom_simgrid = callPackage ./pkgs/simgrid/custom.nix {};
inherit pkgs; # similar to `pkgs = pkgs;` This lets callers use the nixpkgs version defined in this file.
};
in
packages
Similar to the first experiment's default.nix, this file returns a set of packages.
The most significant change is the introduction to the callPackage function, which simplifies the way to call a derivation.
More information about callPackage can be found in the Nix pill about callPackage.
The derivation for the chord simulator has been moved into the pkgs/chord/default.nix file.
Creating a directory for each package is recommended, as it allows to store various versions of the package in the same directory.
callPackage takes two arguments: A path to a Nix file (or to a directory into which it will look for a default.nix file), and a set used to override the inputs of the called package.
Here is the content of the chord package.
{ stdenv, fetchgit, cmake, simgrid, boost }:
stdenv.mkDerivation rec {
pname = "chord";
version = "0.1.0";
src = fetchgit {
url = "https://gitlab.inria.fr/nix-tutorial/chord-tuto-nix-2022";
rev = "069d2a5bfa4c4024063c25551d5201aeaf921cb3";
sha256 = "sha256-MlqJOoMSRuYeG+jl8DFgcNnpEyeRgDCK2JlN9pOqBWA=";
};
buildInputs = [
cmake
simgrid
boost
];
}
Instead of taking the whole pkgs as input as in first experiment's default.nix, this file finely specifies its inputs.
This is done on the first line, which here defines these inputs: stdenv, fetchurl, cmake, simgrid and boost.
Please note that for pkgs/chord/default.nix taken in isolation, these inputs are not defined as they do not have a default value.
This file is meant to be used with callPackage, whose role is to call the package by giving it the requested inputs.
This structure makes it easy to define variations of a package.
This is for example done in the chord_custom_sg attribute of package repository's /default.nix, which uses a custom version of SimGrid instead of the one defined in Nixpkgs.
The custom SimGrid version here is very similar to the one defined in Nixpkgs, as we just wanted to change the SimGrid commit to use.
For more information on package overriding, please refer to the Nix pill on overriding packages and to the Nix documentation on overriding.
{ simgrid, fetchFromGitLab } :
simgrid.overrideAttrs(oldAttrs: rec {
version = oldAttrs.version + "-custom";
src = fetchFromGitLab {
domain = "framagit.org";
owner = "simgrid";
repo = "simgrid";
rev = "fbd3494dc9a7b377cccbc749586313d0f75c15cd";
sha256 = "sha256-qr/ocxlxMw/UXKAkr1puirW6sttwvmjrE1pH/PIAJF4=";
};
})
Usage¶
Now that the package repository is set up, it can be used directly from nix-build or nix-shell.
For example, the following commands builds the chord package defined in the package repository.
nix-build https://gitlab.inria.fr/nix-tutorial/packages-repository/-/archive/master/packages-repository-master.tar.gz -A chord
Alternatively, the package repository can of course be fetched locally manually first.
git clone https://gitlab.inria.fr/nix-tutorial/packages-repository.git /tmp/packages-repository
nix-build /tmp/packages-repository -A chord
Package an experiment using the tiny package repository¶
Defining an experiment that uses the tiny package repository we have just defined is very similar to what we did in the previous experiment.
This is done in the second-experiment directory of the Chord experiments git repository.
Here are commands to retrieve a copy of the experiment and to move into the experiment directory.
git clone https://gitlab.inria.fr/nix-tutorial/chord-experiments.git /tmp/chord-experiments
cd /tmp/chord-experiments/second-experiment
Most files are very similar to those of the previous experiment.
The main difference is that we chose here to provide a shell.nix file instead of a default.nix file.
This is because in our special case we only define a single environment, the one to run the experiment.
Here, shell.nix simply imports our tiny package repository and defines a shell that uses the chord package defined in the tiny package repository.
{
tinypkgs ? import (fetchTarball {
url = "https://gitlab.inria.fr/nix-tutorial/packages-repository/-/archive/master/packages-repository-8e43243635cd8f28c7213205b08c12f2ca2ac74d.tar.gz";
sha256 = "sha256:09l2w3m1z0308zc476ci0bsz5amm536hj1n9xzpwcjm59jxkqpqa";
}) {}
}:
tinypkgs.pkgs.mkShell rec {
buildInputs = [
tinypkgs.chord
];
}
The shebang runner have been slightly adapted to use shell.nix.
#!/usr/bin/env nix-shell
#!nix-shell shell.nix -i bash
chord cluster_backbone.xml s4u-dht-chord_d.xml