nix-snapshotter

README

nix-snapshotter

Brings native understanding of Nix packages to containerd.

Key features • Getting started • Installation • Architecture • Contributing

Key features

• Rather than download image layers, packages come directly from the Nix store.
• Packages can be fetched from a binary cache or built on the fly if necessary.
• Backwards compatible with existing non-Nix images.
• Nix layers can be interleaved with normal layers.
• Allows Kubernetes to resolve image manifests from Nix too.
• Enables fully-declarative Kubernetes resources, including image specification.
• Run pure Nix images without a Docker Registry at all, if you wish.

Getting started

The easiest way to try this out is run a NixOS VM with everything pre-configured.

[!NOTE] You'll need Nix installed with flake support and unified CLI enabled, which comes pre-enabled with Determinate Nix Installer.

Trying without Nix installed

If you have docker or another OCI runtime installed, you can run docker run --rm -it nixpkgs/nix-flakes:

nix run github:pdtpartners/nix-snapshotter#vm

nix run "github:pdtpartners/nix-snapshotter#vm"
nixos login: root # (Ctrl-a then x to quit)
Password: root

# Running `pkgs.hello` image with nix-snapshotter
nerdctl run ghcr.io/pdtpartners/hello

# Running `pkgs.redis` image with kubernetes & nix-snapshotter
kubectl apply -f /etc/kubernetes/redis/

# Wait a few seconds... 
watch kubectl get pods

# And a kubernetes service will be ready to forward port 30000 to the redis
# pod, so you can test it out with a `ping` command
redis-cli -p 30000 ping

Or you can try running in rootless mode:

nix run "github:pdtpartners/nix-snapshotter#vm-rootless"
nixos login: rootless # (Ctrl-a then x to quit)
Password: rootless

# `nerdctl run` with rootless k3s containerd currently not supported yet
# See: https://github.com/containerd/nerdctl/issues/2831
#
# If rootless kubernetes not needed, `nerdctl run` does work with rootless
# containerd + nix-snapshotter.

# Running `pkgs.redis` image with kubernetes & nix-snapshotter
kubectl apply -f /etc/kubernetes/redis/

# Wait a few seconds... 
watch kubectl get pods

# And a kubernetes service will be ready to forward port 30000 to the redis
# pod, so you can test it out with a `ping` command
redis-cli -p 30000 ping

Installation

NixOS and Home Manager modules are provided for easy installation.

[!IMPORTANT] Requires at least nixpkgs 23.05+

• Home Manager

  Flake

  {
    inputs = {
      nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
      home-manager = {
        url = "github:nix-community/home-manager";
        inputs.nixpkgs.follows = "nixpkgs";
      };
      nix-snapshotter = {
        url = "github:pdtpartners/nix-snapshotter";
        inputs.nixpkgs.follows = "nixpkgs";
      };
    };
  
    outputs = { nixpkgs, home-manager, nix-snapshotter, ... }: {
      homeConfigurations.myuser = home-manager.lib.homeManagerConfiguration {
        pkgs = import nixpkgs { system = "x86_64-linux"; };
        modules = [
          {
            home = {
              username = "myuser";
              homeDirectory = "/home/myuser";
              stateVersion = "23.11";
            };
  
            programs.home-manager.enable = true;
  
            # Let home-manager automatically start systemd user services.
            # Will eventually become the new default.
            systemd.user.startServices = "sd-switch";
          }
          ({ pkgs, ... }: {
            # (1) Import home-manager module.
            imports = [ nix-snapshotter.homeModules.default ];
  
            # (2) Add overlay.
            nixpkgs.overlays = [ nix-snapshotter.overlays.default ];
  
            # (3) Enable service.
            virtualisation.containerd.rootless = {
              enable = true;
              nixSnapshotterIntegration = true;
            };
            services.nix-snapshotter.rootless = {
              enable = true;
            };
  
            # (4) Add a containerd CLI like nerdctl.
            home.packages = [ pkgs.nerdctl ];
          })
        ];
      };
    };
  }
  

  Non-flake

  { pkgs, ... }:
  let
    nix-snapshotter = import (
      builtins.fetchTarball "https://github.com/pdtpartners/nix-snapshotter/archive/main.tar.gz"
    );
  
  in {
    imports = [
      # (1) Import home-manager module.
      nix-snapshotter.homeModules.default
    ];
  
    # (2) Add overlay.
    nixpkgs.overlays = [ nix-snapshotter.overlays.default ];
  
    # (3) Enable service.
    virtualisation.containerd.rootless = {
      enable = true;
      nixSnapshotterIntegration = true;
    };
    services.nix-snapshotter.rootless = {
      enable = true;
    };
  
    # (4) Add a containerd CLI like nerdctl.
    home.packages = [ pkgs.nerdctl ];
  }
  

• NixOS

  Flake

  {
    inputs = {
      nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
      nix-snapshotter = {
        url = "github:pdtpartners/nix-snapshotter";
        inputs.nixpkgs.follows = "nixpkgs";
      };
    };
  
    outputs = { nixpkgs, nix-snapshotter, ... }: {
      nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./hardware-configuration.nix
          ({ pkgs, ... }: {
            # (1) Import nixos module.
            imports = [ nix-snapshotter.nixosModules.default ];
  
            # (2) Add overlay.
            nixpkgs.overlays = [ nix-snapshotter.overlays.default ];
  
            # (3) Enable service.
            virtualisation.containerd = {
              enable = true;
              nixSnapshotterIntegration = true;
            };
            services.nix-snapshotter = {
              enable = true;
            };
  
            # (4) Add a containerd CLI like nerdctl.
            environment.systemPackages = [ pkgs.nerdctl ];
          })
        ];
      };
    };
  }
  

  Non-flake

  { pkgs, ... }:
  let
    nix-snapshotter = import (
      builtins.fetchTarball "https://github.com/pdtpartners/nix-snapshotter/archive/main.tar.gz"
    );
  
  in {
    imports = [
      ./hardware-configuration.nix
      # (1) Import nixos module.
      nix-snapshotter.nixosModules.default
    ];
  
    # (2) Add overlay.
    nixpkgs.overlays = [ nix-snapshotter.overlays.default ];
  
    # (3) Enable service.
    virtualisation.containerd = {
      enable = true;
      nixSnapshotterIntegration = true;
    };
    services.nix-snapshotter = {
      enable = true;
    };
  
    # (4) Add a containerd CLI like nerdctl.
    environment.systemPackages = [ pkgs.nerdctl ];
  }
  

• Manual

  See the manual installation docs.

Usage

See package.nix for the Nix interface. You can also repeat the asciinema demo above in examples/declarative-k8s.nix.

pkgs = import nixpkgs {
  overlays = [ nix-snapshotter.overlays.default ];
};

# Builds a native Nix image but intended for an OCI-compliant registry.
redis = pkgs.nix-snapshotter.buildImage {
  name = "ghcr.io/pdtpartners/redis";
  tag = "latest";
  config.entrypoint = [ "${pkgs.redis}/bin/redis-server" ];
};

# Running "${redis.copyToRegistry {}}/bin/copy-to-registry" will copy it to
# an OCI-compliant Registry. It will try to use your Docker credentials to push
# if the target is DockerHub.

# Builds a native Nix image with a special image reference. When running
# the kubelet with `--image-service-endpoint` pointing to nix-snapshotter, then
# it can resolve the image reference to this Nix package.
redis' = pkgs.nix-snapshotter.buildImage {
  name = "redis";
  resolvedByNix = true;
  config.entrypoint = [ "${pkgs.redis}/bin/redis-server" ];
};

# Fully declarative Kubernetes Pod, down to the image specification and its
# contents.
redisPod = pkgs.writeText "redis-pod.json" (builtins.toJSON {
  apiVersion = "v1";
  kind = "Pod";
  metadata = {
    name = "redis";
    labels.name = "redis";
  };
  spec.containers = [{
    name = "redis";
    args = [ "--protected-mode" "no" ];
    image = "nix:0${redis'}";
    ports = [{
      name = "client";
      containerPort = 6379;
    }];
  }];
});

[!NOTE] If you want to understand how nix:0 gets resolved, take a look at the docs for Image Service.

Contributing

Pull requests are welcome for any changes. Consider opening an issue to discuss larger changes first to get feedback on the idea.

Please read CONTRIBUTING for development tips and more details on contributing guidelines.

FAQ

[!IMPORTANT] To understand how it works behind the scenes, see the Architecture docs for more details.

1. What's the difference between this and pkgs.dockerTools.buildImage?

Answer

The upstream buildImage streams Nix packages into tarballs, compresses them and pushes them to an OCI registry. Since there is a limit to number of layers in an image, a heuristic is used to put popular packages together. There is large amount of duplication between your Nix binary cache and the Docker Registry tarballs, and even between images that share packages as the layers may duplicate common packages due to the heuristic-based layering strategy.

With pkgs.nix-snapshotter.buildImage, containerd natively understand Nix packages, so everything is pulled at package granularity without the layer limit. This means all the container content is either already in your host nix store or fetched from your Nix binary cache.

1. What's the difference between this and Nixery?

Answer

Nixery exposes an API (in the form of an OCI registry) to dynamically build Nix-based images. It has an improved layering design compared to upstream pkgs.dockerTools.buildImage but is still fundamentally a heuristics- based layering strategy (see above), so it still suffers from the same inefficiency in duplication. However, Nixery can totally start building nix-snapshotter images so we can have a Docker Registry that can dynamically build native Nix images. See this Nixery issue to follow along the progress.

1. What's the difference between this and a nix-in-docker?

Answer

If you run nix inside a container (e.g. nixos/nix or nixpkgs/nix-flake) then you are indeed fetching packages using the Nix store. However, each container will have its own Nix store instead of de-duplicating at the host level.

nix-snapshotter is intended to live on the host system (sibling to containerd and/or kubelet) so that multiple containers running different images can share the underlying packages from the same Nix store.

1. What's the difference between this and nix2container?

Answer

nix2container improves upon pkgs.dockerTools.buildImage in a few ways. First it does something similar to pkgs.dockerTools.streamLayeredImage where it avoids writing Nix layer tarballs to Nix store and builds them JIT when exporting, like with it's passthru attribute copyToRegistry. This avoids writing Nix layer tarballs into the Nix store unnecessarily.

Secondly, it separates out image metadata and layer metadata. This means that when updating the image config, layers don't need to be rebuilt. Thirdly, each layer metadata is in its own Nix package, so only updated layers need to be rebuilt.

Lastly, the layer metadata is a JSON that contains the Nix store paths along with the digest which is computed from the layer tarball which is thrown away. This lets the tool skopeo to only copy non-existing layers, which then builds the requested layer tarballs again JIT.

nix2container is a great improvement, but still suffers same problems pointed out in the pkgs.dockerTools.buildImage section. It duplicates data between Nix binary cache and Docker Registry, and it duplicates packages between layers due to using a similar heuristic-based strategy.

pkgs.nix-snapshotter.buildImage has all the same improvements, except that we do write the final image back to the Nix store since it's tiny and allows us to resolve image manifests via a Nix package.

License

The source code developed for nix-snapshotter is licensed under MIT License.

This project also contains modified portions of other projects that are licensed under the terms of Apache License 2.0. See NOTICE for more details.