The Nix lectures, part 2: Derivations

This is part 2 of a tutorial series that covers all about Nix. We covered language basics in the first part, and in this post we will cover derivations.

• Part 1: Introduction and language basics
• Part 2: Derivations
• Part 3: NixOS and module system (coming soon)
• Part 4: Integration (coming soon)

In chapter 1, we gave a high-level overview of the Nix project. In short, packages are defined in Nix, a purely functional programming language. We use the nix program to evaluate expressions.

We covered the very basic types of the Nix language, such as strings or attrsets. A derivation is yet another type, that can also be built.

builtins.derivation

Derivations are the privimite that Nix uses to define packages. “Package” is a loosely defined term, but a derivation is simply the result of calling builtins.derivation. Because derivations are very cheap to create, in Nix we tend to use a derivation for anything — like configuration files. Would you call a config.yaml a package? Probably not, but it can be built as a derivation.

builtins.derivation is a function that takes an attrset with some known keys, and the rest are set as environment variables. You can check the reference documentation in the nix manual. A simple invocation might look like this:

# my-derivation.nix
builtins.derivation {
  system = "x86_64-linux";
  name = "sample";
  builder = "/bin/sh";

  args = [
    "-c"
    "echo hello > $out"
  ];
}
#=> «derivation /nix/store/n34150nf03sh04j8mjzm8sawdqx9sgqi-sample.drv»

A derivation runs the builder with some args. The environment variable $out is where the package results must be placed. Instead of installing to /usr/bin, you install to $out/bin, etc.

Derivations can be built in the repl with :b, or with the command nix build -f my-derivation.nix.

Nix will tell use that the return type is a derivation. However, from chapter 1, I didn’t mention that this type exists. What is happening here is that derivations are implemented as attrsets with a special field called type.

{ type = "derivation";  }
#=> «derivation»

Derivations are attrsets, which means we can:

• Access fields from derivations, like pkgs.hello.overrideAttrs.
• Do string interpolation because they have an outPath field.

pkgs.hello
#=> «derivation /nix/store/crmj28zg09517n5sskml9fmy2c6r3rsr-hello-2.12.1.drv»

pkgs.hello.outPath
#=> "/nix/store/yb84nwgvixzi9sx9nxssq581pc0cc8p3-hello-2.12.1"

"here is ${pkgs.hello}"
#=> "here is /nix/store/yb84nwgvixzi9sx9nxssq581pc0cc8p3-hello-2.12.1"

# two-derivations.nix
let
  first = builtins.derivation {
    system = "x86_64-linux";
    builder = "/bin/sh";
    name = "first";
    args = ["-c" "echo first > $out"];
  };

  second = builtins.derivation {
    system = "x86_64-linux";
    builder = "/bin/sh";
    name = "second";
    args = [
      "-c"
      ''echo "${first}" > $out''
    ];
  };
in
  second
#=> «derivation /nix/store/cbk704h7sffj57hmrp5fm2cs6xpy15nq-second.drv»

$ nix build -f ./two-derivations.nix --dry-run
these 2 derivations will be built:
  /nix/store/fasbn3837frywijgjnj33qr51018pyn3-first.drv
  /nix/store/cbk704h7sffj57hmrp5fm2cs6xpy15nq-second.drv

This example shows that string interpolation can be used to refer to other derivation programmatically, because Nix tracks the context through different values. Dependencies are handled automatically for you.

Fixed-output derivations

To be useful, we need Nix to download source code from the internet. As any sane build system, Nix also implements a sandbox that is used during builds. This sandbox exposes:

• /nix/store
• “Private” versions of /proc, /dev, etc.
• Private PID, mount namespaces.
• /bin/sh — many scripts rely on it existing.
• ❗ No internet access

So, how do we handle downloading the source tarballs of projects? In the Nix “purity” model, we find the concept of a fixed-output derivation, or FOD for short.

A fixed-output-derivation disables the internet sandbox, such that we can do any http connection normally. However, the user must provide a hash of the resulting output. If another user tries to build the same FOD, and the hash is correct, it means the network connection produced the same resulting derivation.

# fixed-output.nix
let
  pkgs = import <nixpkgs> {};
in
  builtins.derivation {
    name = "fixed-output";
    system = "x86_64-linux";
    builder = "/bin/sh";

    args = [
      "-c"
      "${pkgs.curl}/bin/curl -H 'User-Agent: nix' https://httpbin.org/user-agent > $out"
    ];

    # fixed-output-derivation by using output* options
    outputHashMode = "flat";
    outputHashAlgo = "sha256";
    outputHash = "";
  }

# $ nix build -f ./fixed-output.nix
# warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA='
# error: hash mismatch in fixed-output derivation '/nix/store/9simizzhrmsq5nnannh8gvgxqm465nwb-fixed-output.drv':
#          specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
#             got:    sha256-3FWguG761jOvjXDVGi2BkN+hLCUurMCBzeF4m0AEr4A=

However, if the resulting derivation doesn’t match the provided hash, Nix will error out. With this system, we allow some “impurity”, at the expense of having to know what the output is.

In contrast, for a regular derivation, we don’t know ahead of time the output hash. But we assume, that for the same inputs, the derivation will produce the same outputs. This model is called “input-addressed”.

Building derivations

Nix is an expression-based language, and every nix file must return a single value. If it returns a derivation, instead of nix eval, you use nix build to tell the nix-daemon to build the derivation for you.

nix build takes an “installable” as an argument, which can be:

• -f [filename.nix] [attribute]
• [flake.nix]#[flake output]
• --expr "literal nix expression"

When using -f, instead of returning a derivation, you can return an attrset of derivations, which you select with the attribute name:

# filename.nix
{
  foo = builtins.derivation { ... };
  bar = builtins.derivation { ... };
}

# $ nix build -f ./filename.nix foo

Some of the flags you should know about are:

• -L or --print-build-logs — to make Nix print the build logs.
• --keep-failed — keep the build directory, if the derivation fails to build.

stdenv

While builtins.derivation is the low-level primitive, 99% of your time will be spent using the higher-level abstractions from nixpkgs.

The standard environment (stdenv for short) is a collection of build utilities that makes builtins.derivation nicer to use. You can find stdenv in nixpkgs, the package collection of Nix. Note that Nix the program and nixpkgs are “distributed” separately.

We can access the stdenv from a pkgs instance. Pkgs is by convention the result of calling import /path/to/nixpkgs {}: importing nixpkgs and evaluating with some configuration.

• nixpkgs -> the package repository, a path
• pkgs -> result of import /path/to/nixpkgs {}, an attribute set

Pkgs is a huge attrset that contains all of the packages, as name-value pairs. One of those is pkgs.stdenv:

let
  # <nixpkgs> is set-up by a regular nix installation
  pkgs = import <nixpkgs> {};
in
  pkgs.stdenv
#=> «derivation /nix/store/sbxnap725qcfsjgp21vjd1h6qpgi6gwj-stdenv-linux.drv»

mkDerivation basics

stdenv.mkDerivation is a function that takes an attrset and returns a derivation. The arguments of mkDerivation, as with builtins.derivation are exported as environment variables. The stdenv builder, a bash script, interprets the environment variables to build your package.

A mkDerivation call will contain the following:

• name or pname + version of the package. The latter is preferred.
• src can be another derivation or a local path.
• Dependencies: packages that our package depends on, at run-time or build-time.
• Phases: pieces of bash code.
• Any other environment variable that is used by the build scripts.

mkDerivation heavily relies on bash code, and environment variables. As all the attributes of the argument will be mapped into environment variables, we use Nix as a “fancy bash writer”.

To begin with, name is something that is passed-through to builtins.derivation. You can choose to split it with pname and version, which is very convenient — pname stands for package name.

As mentioned, src will be mapped into the environment variable $src. The bash builder has a phase that takes $src and tries to unpack it into the work directory. We can use anything that makes sense where a path does, such as another derivation.

Phases

mkDerivation is implemented as a bash script. It has different “phases” that you can expect from a build system: build, test, install, etc. You can modify the behaviour of the phases with environment variables, or you can choose to override them entirely.

The phases are documented in the nixpkgs manual and are the following:

• unpackPhase unpacks whatever you put in src
• patchPhase applies the patches you give with patches = [ ./my.patch ]
• configurePhase runs ./configure for autotools-based packages.
• ⭐ buildPhase builds the package
• checkPhase runs tests
• ⭐ installPhase moves files into $out
• fixupPhase Nix-specific sanity checks, like setting proper store paths
• installCheckPhase runs tests that check if the package is installed properly
• distPhase — I have never seen this one in use

You can choose to overwrite the default phases. The default implementation of the phases also run hooks before and after, to not have to completely modify a phase.

let
  pkgs = import <nixpkgs> {};
in
  pkgs.stdenv.mkDerivation {
    # ...
    patches = [
      ./my.patch
    ];

    postPatch = ''
      # this runs after the standard patchPhase, which applies `patches`
    '';

    buildPhase = ''
      # completely change the build phase
    '';

    postBuild = ''
      # this won't run, because we replace buildPhase
      # you can run this by calling `runHook postBuild` (and `runHook preBuild`)
      # in your custom buildPhase
    '';
  }

Many phases also have some configuration. For example, you can control unpackPhase by setting the env variable dontUnpack = true, or skipping the checks with doCheck = false. Refer to the manual for specifics of each phase.

The default behavior of the stdenv for buildPhase and installPhase is make + make install. If you have a simple C program, then you don’t need anything else. But you probably don’t, so let’s keep dissecting stdenv.

Dependencies

The different types of dependencies can be a complex topic. It gets messy when handling cross-compilation, because you must not mix the different architecures (e.g. using an x86_64 GCC to build an ARM64 binary).

In Nix, you will use 2 kind of dependencies:

• nativeBuildInputs — packages that are needed on the build machine.
  • For example: GCC, make, pkg-config, etc (usually programs).
• buildInputs — packages that are needed on the machine that runs the package.
  • For example: libc, QT, zlib, etc (usually libraries).

By default, Nix doesn’t check that the dependencies do respect this difference, unless you use strictDeps = true. However, hooks only run for packages in nativeBuildInputs.

Hooks

One of the difference of nativeBuildInputs, is that they may contain some bash code that is sourced by the bash builder. Having a package with a hook in buildInputs won’t trigger its stdenv hook.

Hooks augment the capabilities of mkDerivation, depending on the package. Usually, they are executed for the program to be able to function properly. This is best shown by using the example of the hook of pkg-config.

pkg-config is a program that allows us to find the path to libraries. This is very nice in Nix, because there is no global library path. However, how do we transmit to pkg-config, that the packages in buildInputs are available? The hook is what “glues” everything together. For pkg-config, its hook will export PKG_CONFIG_PATH, which is used by the CLI:

# pkg-config.nix
let
  pkgs = import <nixpkgs> {};
in
  pkgs.stdenv.mkDerivation {
    name = "with-pkg-config";
    src = null;
    dontUnpack = true;

    nativeBuildInputs = [
      pkgs.pkg-config
    ];

    buildInputs = [
      pkgs.zlib
    ];

    buildPhase = ''
      echo "PKG_CONFIG_PATH:$PKG_CONFIG_PATH"
      pkg-config --list-all
      printenv
    '';
  }

$ nix build -f ./pkg-config.nix -L
with-pkg-config> PKG_CONFIG_PATH:/nix/store/0w4q8asq9sn56dl0sxp1m8gk4vy2ygs8-zlib-1.3.1-dev/lib/pkgconfig
with-pkg-config> zlib zlib - zlib compression library

Compiler wrappers

Apart from manually passing the include flags from pkg-config, in Nix we also use the C/C++ compiler wrappers. In general, a wrapper is a “fake” application that calls the real one, passing some extra arguments.

In nixpkgs, gcc (or clang in macOS) is a bash script. It sets some flags, and then calls the real gcc.

$ file $(which gcc)
/nix/store/...-gcc-wrapper-13.2.0/bin/gcc: a /nix/store/...-bash-5.2p26/bin/bash script, ASCII text executable

$ tail -n6 $(which gcc)
else
    exec /nix/store/14c6s4xzhy14i2b05s00rjns2j93gzz4-gcc-13.2.0/bin/gcc \
       ${extraBefore+"${extraBefore[@]}"} \
       ${params+"${params[@]}"} \
       ${extraAfter+"${extraAfter[@]}"}
fi

The stdenv sets up some flags like NIX_CFLAGS_COMPILE or NIX_LDFLAGS derived from buildInputs. This makes a “fake global search path”, that are passed to the wrapped gcc. So, if you use -lz, gcc will find the library normally, because the wrapper knows where to find the libraries — instead of in /usr.

# cflags.nix
let
  pkgs = import <nixpkgs> {};
in
  pkgs.stdenv.mkDerivation {
    name = "cflags";
    src = null;
    dontUnpack = true;

    buildInputs = [ pkgs.zlib ];

    buildPhase = ''
      printenv | grep NIX_ | grep FLAGS
    '';
  }

$ nix build -f ./cflags.nix -L
cflags> NIX_CFLAGS_COMPILE= -frandom-seed=4sm1s48afi -isystem /nix/store/0w4q8asq9sn56dl0sxp1m8gk4vy2ygs8-zlib-1.3.1-dev/include -isystem /nix/store/0w4q8asq9sn56dl0sxp1m8gk4vy2ygs8-zlib-1.3.1-dev/include
cflags> NIX_LDFLAGS=-rpath /nix/store/4sm1s48afihxh46mdv71rcwva52w47hm-cflags/lib  -L/nix/store/rqs1zrcncqz3966khjndg1183cpdnqxs-zlib-1.3.1/lib -L/nix/store/rqs1zrcncqz3966khjndg1183cpdnqxs-zlib-1.3.1/lib

Modifying derivations

A very common task in Nix is taking a package, and modifying some of its values. Usually, you want to upgrade some package to a newer version, or pin it to an older one. There are multiple ways to do it, which will be explained in the following sections.

overrideAttrs

When you use mkDerivation, the resulting derivation will contain an extra attribute named .overrideAttrs. This is a function, to which you can pass a function such as oldAttrs: newAttrs. This means that we change the attributes of the mkDerivation call, and optionally use the older ones if we need to. For example, we can use .overrideAttrs to change the name of the hello package:

let
  pkgs = import <nixpkgs> {};
in
  pkgs.hello.overrideAttrs (old: {
    pname = "goodbye";
  })
#=> «derivation /nix/store/0b00bv1hjv7v1dlnzgrv21caz4h7ihii-goodbye-2.12.1.drv»

A common task would be to generate a new package that changes the version to an updated one, or using a different fork of the project.

• Change the src to a different one
• Optionally change version or pname. Note that these are only useful to humans, Nix doesn’t use them.
• If the build process changed, adapt the package by tweaking the different phases.
• Optionally, change ony other attributes, like patches or buildInputs

let
  pkgs = import <nixpkgs> {};
in
  pkgs.hello.overrideAttrs (old: {
    # using the old attributes
    pname = old.pname + "-mod";

    # changing to a different version/fork
    src = pkgs.fetchFromGitHub { ... };

    # adjusting your dependencies
    buildInputs = old.buildInputs ++ [ pkgs.zlib ];

    postInstall = ''
      # run extra code after installation, etc
    '';
  })

override and callPackage pattern

Overriding a package can be very nuanced. In the previous example, it can be hard to control which dependencies are used.

Let’s imagine that pkgs.hello depended on pkgs.zlib. How could we change which pkgs.zlib is used, only with .overrideAttrs? You could imagine doing something like:

.overrideAttrs (old: {
  # we don't do this
  buildInputs =
    (builtins.filter (drv: (lib.getName drv) != "zlib") old.buildInputs)
    ++
    [ newZlib ];
})

That thing “works”, but for more nuanced cases it may be impossible. It is the case if instead of using buildInputs, we do some string interpolation, or any “non-structured” use of the dependencies.

The solution that nixpkgs proposes is the callPackage pattern. pkgs.callPackage file args takes a path to a file which must contain a function, and calls the function with pkgs // args — a merged pkgs with our own modifications.

# package.nix
# function that returns a derivation
{
  stdenv,
  zlib,
}: stdenv.mkDerivation {
  name = "foo";
  buildPhase = ''
    echo ${zlib} > $out
  '';
  # ...
}

# default.nix
let
  pkgs = import <nixpkgs> {};
in
  # calls package.nix with pkgs.stdenv and pkgs.zlib automatically
  pkgs.callPackage ./package.nix {}

After using callPackage, our derivation will have an extra attribute called .override. We can use .override, to go “back in time” and call the original file with different arguments. This is very powerful, because we can replace the dependencies of a package with a simple .override, and every usage in the file will follow the new package.

In the following example, we create a custom zlib, and we use callPackage and .override to pass it to a package:

# default.nix
let
  pkgs = import <nixpkgs> {};

  myPkg = pkgs.callPackage ./package.nix {
    zlib = pkgs.zlib.overrideAttrs ...;
  };

  # we can override pkgs from nixpkgs too
  myHello = pkgs.hello.override {
    stdenv = ...;
  };
in
  # ...

Package collections

In the previous sections we have learn:

• The basics of derivations.
• Using mkDerivation as an abstraction.
• Modifying packages with .overrideAttrs.
• Using callPackage and .override for dependency injection.

The next step, is learning how to handle a collection of different packages, and threading the dependencies between them. We will start with a simple example: we have 3 packages that depend on each other:

• myLib
• simpleCLI, that depends on myLib
• complexCLI, that depends on myLib and simpleCLI

We will use pkgs.callPackage which passes pkgs to the arguments. But, because these 3 new packages are not in nixpkgs, we will need to pass the explicitely:

let
  pkgs = import <nixpkgs> {};

  myLib = pkgs.callPackage ./package-mylib.nix {};
  simpleCLI = pkgs.callPackage ./package-simplecli.nix {
    inherit myLib;
  };
  complexCLI = pkgs.callPackage ./package-complexcli.nix {
    inherit myLib simpleCLI;
  };
in
  # ...

This manual “threading” of dependencies is fine for 1 or 2 packages, but it will become annoying quickly. To solve this, there are two different solutions: callPackageWith and overlays.

callPackageWith

As you know, pkgs.callPackage file args will call the file with (pkgs // args). callPackageWith is simply a way to define what we want to call the packages with, as the name implies.

First, you define your own custom callPackage:

myCallPackage = pkgs.callPackageWith (pkgs // mypkgs);

And then, simply use it to define your package set. It may look like an infinite-recursion or self-referencing problem, but Nix can deal with it no problem. All your dependencies will be threaded automatically with your custom callPackage.

let
  pkgs = import <nixpkgs> {};
  myCallPackage = pkgs.callPackageWith (pkgs // mypkgs);

  mypkgs = {
    myLib = myCallPackage ./package-mylib.nix {};
    simpleCLI = myCallPackage ./package-simplecli.nix {}; # myLib passed automatically
    complexCLI = myCallPackage ./package-complexcli.nix {}; # " and simpleCLI passed too
  };
in
  # ...

{
  stdenv,
  myLib,
  simpleCLI,
}:
stdenv.mkDerivation {
  name = "mypackage";
  nativeBuildInputs = [ simpleCLI ];
  buildInputs = [ myLib ];
  # ...
}

overlays

Overlays is one of the “buzzwords” that you may have heard from Nix. But if you have reached here, then you have all the knowledge to understand what overlays are used for.

Similarly to the callPackageWith pattern described before, overlays allow to inject our package collection into pkgs itself. Instead of using a custom callPackage that threads our custom packages, we are “modifying” pkgs itself.

Overlays are loaded when importing nixpkgs as configuration. We load a list of overlays, and an overlay is a function that takes two arguments: the package set previous to the overlay, and the final package set.

final: prev: {
  myLib = final.callPackage ./package-mylib.nix {};

  # myLib passed automatically from final
  # because it's on final.myLib
  simpleCLI = final.callPackage ./package-simplecli.nix {};
  complexCLI = final.callPackage ./package-complexcli.nix {};

  # you can also modify pkgs from nixpkgs
  # this will be final.zlib, so modify zlib from prev
  zlib = prev.zlib.overrideAttrs ....;
}

Both prev and final are like pkgs — that is, a huge attrset containing all the packages. prev is pkgs before applying your overlay. final is pkgs after applying all overlays, including the one you are writing.

There is one weird thing here: the result of applying the overlay is an input to the overlay itself. This is possible because Nix is a lazy language, but you can also run into infinite recursion problems.

When should you grab something from prev and when from final? The rule of thumb is the following:

• Take from prev what you want to modify, e.g. pkg = prev.pkg.overrideAttrs ...
• Take from final otherwise. As it will account for every modification from every package. We use final.callPackage to thread the modifications of our overlay.

To load an overlay, you must pass it as configuration when importing nixpkgs:

let
  pkgs = import <nixpkgs> {
    overlays = [
      (final: prev: {
        myLib = final.callPackage ./package-mylib.nix {};
      })
    ];
  };
in
  pkgs.myLib

Trivial builders and writers

There are higher-level abstractions built on top of stdenv.mkDerivation, that can be very useful. To begin with, the trivial builders implement very basic tasks:

• runCommand, when all you need is some bash script to write to $out

  pkgs.runCommand "name" {} ''
    echo "hello" > $out
  ''
  

• writeText, that puts a string into $out

  pkgs.writeText "name" "hello"
  

We also find in nixpkgs the format writers and script writers. The format writers take a Nix value, and transforms it into the target format, placing it in a new file. This is useful to transform nix code into YAML, TOML, etc.

(pkgs.formats.yaml {}).generate "config.yaml" { foo.bar = "baz"; }
# «derivation /nix/store/yplv61sf4vqkmrza8m31b1808wz090fw-config.yaml.drv»

$ nix build -f ./some-yaml.nix

$ cat result
foo:
  bar: baz

The script writers can be used to quickly create scripts or utilities for our terminal usage:

with import <nixpkgs> {};
  writeShellScriptBin "hello" ''
    res=$(${jq}/bin/jq .something /some/path)
    echo $res
  ''

Fetchers

Fetching the source code of an application is one essential step for a build system like Nix. As with packages, the fetched content will be placed in /nix/store, but we have 2 ways of doing so:

• Eval-time fetchers: builtins.fetch{*}. Used for downloading Nix code that we can evaluate.
• Build-time fetchers: pkgs.fetch{*}. Only used at build-time for other derivations. They produce derivations.

We will use eval-time fetcher when we use Nix code from them. Nix is able to download the eval-time fetchers as it evaluates the code, but it can only do so in a sequence. Usually, we only need to fetch nixpkgs at eval-time:

let
  # pinning nixpkgs to a remote tarball
  # we use a builtins fetcher because it runs at evaluation-time
  myNixpkgs = builtins.fetchTarball "https://github.com/NixOS/nixpkgs/archive/06278c77b5d162e62df170fec307e83f1812d94b.tar.gz";
  pkgs = import myNixkgs {};
in
  # ...

Build-time fetchers produce regular fixed-output derivations. They use regular programs like curl or git clone under the hood. As FOD’s, you have to provide the output hash. The fetchers are documented in nixpkgs.

let
  pkgs = import <nixpkgs> {};
in
  pkgs.fetchFromGitHub {
    owner = "viperML";
    repo = "neohome";
    rev = "<rev>";
    hash = "<hash>";
  }

Remember to:

• Use an eval-time fetcher to fetch nixpkgs, or any nix code you want to evaluate.
• Use the build-time fetchers from pkgs for the rest.

{
  stdenv,
}:
stdenv.mkDerivation {
  # Bad: this is builtins.fetchurl, it's defined in the global scope
  src = fetchurl { /* ... */ };
}

{
  stdenv,
  fetchurl,
}:
stdenv.mkDerivation {
  # Good: this is pkgs.fetchurl, the proper build-time fetcher
  src = fetchurl { /* ... */ };
}

Eval-time purity

I also want to add this technical topic about eval-time purity: the eval-time fetchers don’t require an output hash, as FOD’s do. You can call builtins.fetchTarball directly, without providing the output hash.

Originally, this behaviour was intended to be for convenience. However, one of the features that flakes have is that they disallow eval-time fetchers that don’t provide an output hash.

builtins.fetchurl {
  url = "https://github.com/NixOS/nixpkgs/archive/nixos-14.12.tar.gz";
  # OK without flakes

  # if using flakes, passing the sha256 is required
  # sha256 = "...";
}

Language and frameworks

Nixpkgs contains many abstractions built on top of mkDerivation for the specifics of each language. In fact, you will be using mkDerivation only for C/C++ projects, the trivial builders for simple files, and the language-specific builders for the rest.

• To build Python packages, use buildPythonPackage
• To build Go, use buildGoModule
• To build Rust packages, use buildRustPackage
• etc.

In previous sections we have been exploring Nix together. But for the language builders, it is best to consult the nixpkgs documentation.

Nixpkgs manual: Languages and frameworks

In general, instead of using stdenv.mkDerivation, you will use a different function. The arguments will be still similar to mkDerivation, but you will need to configure “extra” stuff. For example, for Rust packages you have to point buildRustPackage to a hash or path of the Cargo.lock, the Python builder will properly wrap the resulting binaries, etc.

Python and other languages also have a huge collection of packaged dependencies from PyPi. These are located in their own custom scope in nixpkgs, for Python that is pkgs.python3.pkgs.<name>. Because of this, you also need to use pkgs.python3.pkgs.callPackage to be able to access the packages from this scope:

let
  pkgs = import <nixpkgs> {};

  myPythonPkg = pkgs.python3.pkgs.callPackage ./myPythonPkg.nix {};
in
  # ...

# myPythonPkg.nix
{
  python3,
  buildPythonPackage,
  requests, # <- now this refers to pkgs.python3.pkgs.requests
}:
buildPythonPackage {
  # ...
}

Shells

Building packages into the /nix/store is OK, but what about the actual development? You don’t want to run nix build for each build iteration, and your editor and tooling must know about your project dependencies.

Enter nix develop. It accepts the same arguments as nix build, and we can test it by passing it a nix file that returns a derivation:

let
  pkgs = import <nixpkgs> {};
in
  pkgs.hello
#=> «derivation /nix/store/crmj28zg09517n5sskml9fmy2c6r3rsr-hello-2.12.1.drv»

$ nix develop -f ./dev.nix

$ gcc --version
gcc (GCC) 13.3.0

Nix just started a new subshell, and exported all the environment variables used for building pkg.hello. This is the last piece, that allows us to use Nix to develop and package software. Within the shell, we will have access to gcc, all our dependencies. The wrapper flags such as NIX_CFLAGS_COMPILE all the hook from pkg-config, cmake and other are also executed.

Using nix develop allows us to define all our dependencies in a Nix file, that can be shared along all the contributors of the project. This is a very important feature of Nix, as it replaces the workflow of running apt install commands that modify your system, or having to deal with containers.

If you don’t want to write a package definition with mkDerivation right away, you can instead use pkgs.mkShell. It is a quick to use alternative, that only adds some dependencies into the environment. It is derivation, meaning you can use nix develop -f shell.nix:

let
  pkgs = import <nixpkgs> {};
in
  pkgs.mkShell {
    packages = [
      pkgs.pkg-config
      pkgs.zlib
      pkgs.python3
    ];

    env = {
      FOO = "BAR";
    };
  }
#=> «derivation /nix/store/dhf2c5g274zy58pdb15cdmwrzs69sqin-nix-shell.drv»

$ nix develop -f ./dev.nix

$ pkg-config --list-all
python            Python - Build a C extension for Python
zlib              zlib - zlib compression library
python3           Python - Build a C extension for Python
python3-embed     Python - Embed Python into an application
python-3.12-embed Python - Embed Python into an application
python-3.12       Python - Build a C extension for Python

$ python3 --version
Python 3.12.5

$ printenv FOO
BAR

mkShell is documented in the nixpkgs manual, and you will commonly use the following options:

• packages list of dependencies. Hooks are run as needed.
• env: attrset of environment variables to export

I have another post that covers the language-specific details for mkShell, if you want to continue reading: Development workflow with Nix shells.

Notes on the Nix V2 vs V3 CLI

At the time of this writing, the last version of Nix is 2.24. There is an ongoing development process to rewrite the command line interface, to make it friendlier and add more features. This is called the Nix V3 CLI, gated behind the experimental feature nix-command.

For the sake of writing a future-proof guide, I chose to show the V3 commands, but here is an small rosetta table:

Finale

I hope that this lesson was useful, and you learned how derivations and packages work in Nix. If you have any question or thing to improve, feel free to open an issue or pull request to the repo of this blog — link in the footer.

In summary, we learned that:

• builtins.derivations is the primitive to create a derivation.
• Derivations are attrsets
• You can string interpolate a derivation "${pkgs.hello}/bin/hello"
• Nixpkgs defines higher level abstractions with stdenv and mkDerivation
• mkDerivation has the concepts of phases, build/run time dependencies and compiler wrappers.
• You can modify existing derivations with .overrideAttrs and .override
• You can define package collections with callPackageWith and overlays
• Language-specific abstractions are built on top of mkDerivation
• mkShell and nix develop can be used to get a development environment for a derivation.

In part 3 of the tutorial, we will be exploring NixOS and the module system.