Devmapper snapshotter

Devmapper is a containerd snapshotter plugin that stores snapshots in ext4-formatted filesystem images in a devicemapper thin pool.


To make it work you need to prepare thin-pool in advance and update containerd's configuration file. This file is typically located at /etc/containerd/config.toml.

Here's minimal sample entry that can be made in the configuration file:

    pool_name = "containerd-pool"
    base_image_size = "8192MB"

The following configuration flags are supported:

  • root_path - a directory where the metadata will be available (if empty default location for containerd plugins will be used)
  • pool_name - a name to use for the devicemapper thin pool. Pool name should be the same as in /dev/mapper/ directory
  • base_image_size - defines how much space to allocate when creating the base device
  • async_remove - flag to async remove device using snapshot GC's cleanup callback

Pool name and base image size are required snapshotter parameters.


Give it a try with the following commands:

ctr images pull --snapshotter devmapper
ctr run --snapshotter devmapper test


The devicemapper snapshotter requires dmsetup (>= 1.02.110) command line tool to be installed and available on your computer. On Ubuntu, it can be installed with apt-get install dmsetup command.

How to setup device mapper thin-pool

There are many ways how to configure a devmapper thin-pool depending on your requirements, disk configuration, and environment.

On local dev environment you can utilize loopback devices. This type of configuration is simple and suits well for development and testing (please note that this configuration is slow and not recommended for production uses). Run the following script to create a thin-pool device:

set -ex


mkdir -p ${DATA_DIR}

# Create data file
sudo touch "${DATA_DIR}/data"
sudo truncate -s 100G "${DATA_DIR}/data"

# Create metadata file
sudo touch "${DATA_DIR}/meta"
sudo truncate -s 10G "${DATA_DIR}/meta"

# Allocate loop devices
DATA_DEV=$(sudo losetup --find --show "${DATA_DIR}/data")
META_DEV=$(sudo losetup --find --show "${DATA_DIR}/meta")

# Define thin-pool parameters.
# See for details.
DATA_SIZE="$(sudo blockdev --getsize64 -q ${DATA_DEV})"

# Create a thin-pool device
sudo dmsetup create "${POOL_NAME}" \

cat << EOF
# Add this to your config.toml configuration file and restart containerd daemon
    pool_name = "${POOL_NAME}"
    root_path = "${DATA_DIR}"
    base_image_size = "10GB"

Use dmsetup to verify that the thin-pool created successfully:

sudo dmsetup ls
devpool	(253:0)

Once configured and restarted containerd, you'll see the following output:

INFO[2020-03-17T20:24:45.532604888Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
INFO[2020-03-17T20:24:45.532672738Z] initializing pool device "dev-pool"

Another way to setup a thin-pool is via container-storage-setup tool (formerly known as docker-storage-setup). It is a script to configure CoW file systems like devicemapper:

set -ex

# Block device to use for devmapper thin-pool

# Install container-storage-setup tool
git clone
cd container-storage-setup/
sudo make install-core
echo "Using version $(container-storage-setup -v)"

# Create configuration file
# Refer to `man container-storage-setup` to see available options
sudo tee /etc/sysconfig/docker-storage-setup <<EOF

# Run the script
sudo container-storage-setup

cat << EOF
# Add this to your config.toml configuration file and restart containerd daemon
    pool_name = "${VG_NAME}-${POOL_NAME}"
    base_image_size = "10GB"

If successful container-storage-setup will output:

+ echo VG=containerd
+ sudo container-storage-setup
INFO: Volume group backing root filesystem could not be determined
INFO: Writing zeros to first 4MB of device /dev/xvdf
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 0.0162906 s, 257 MB/s
INFO: Device node /dev/xvdf1 exists.
  Physical volume "/dev/xvdf1" successfully created.
  Volume group "containerd" successfully created
  Rounding up size to full physical extent 12.00 MiB
  Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  Logical volume "devpool" created.
  Logical volume containerd/devpool changed.

And dmsetup will produce the following output:

sudo dmsetup ls
containerd-devpool          (253:2)
containerd-devpool_tdata    (253:1)
containerd-devpool_tmeta    (253:0)
Expand ▾ Collapse ▴




This section is empty.


View Source
var (
	// ErrNotFound represents an error returned when object not found in meta store
	ErrNotFound = errors.New("not found")
	// ErrAlreadyExists represents an error returned when object can't be duplicated in meta store
	ErrAlreadyExists = errors.New("object already exists")


This section is empty.


type Config

type Config struct {
	// Device snapshotter root directory for metadata
	RootPath string `toml:"root_path"`

	// Name for 'thin-pool' device to be used by snapshotter (without /dev/mapper/ prefix)
	PoolName string `toml:"pool_name"`

	// Defines how much space to allocate when creating base image for container
	BaseImageSize      string `toml:"base_image_size"`
	BaseImageSizeBytes uint64 `toml:"-"`

	// Flag to async remove device using Cleanup() callback in snapshots GC
	AsyncRemove bool `toml:"async_remove"`

    Config represents device mapper configuration loaded from file. Size units can be specified in human-readable string format (like "32KIB", "32GB", "32Tb")

    func LoadConfig

    func LoadConfig(path string) (*Config, error)

      LoadConfig reads devmapper configuration file from disk in TOML format

      func (*Config) Validate

      func (c *Config) Validate() error

        Validate makes sure configuration fields are valid

        type DeviceInfo

        type DeviceInfo struct {
        	// DeviceID is a 24-bit number assigned to a device within thin-pool device
        	DeviceID uint32 `json:"device_id"`
        	// Size is a thin device size
        	Size uint64 `json:"size"`
        	// Name is a device name to be used in /dev/mapper/
        	Name string `json:"name"`
        	// ParentName is a name of parent device (if snapshot)
        	ParentName string `json:"parent_name"`
        	// State represents current device state
        	State DeviceState `json:"state"`
        	// Error details if device state change failed
        	Error string `json:"error"`

          DeviceInfo represents metadata for thin device within thin-pool

          type DeviceInfoCallback

          type DeviceInfoCallback func(deviceInfo *DeviceInfo) error

            DeviceInfoCallback is a callback used for device updates

            type DeviceState

            type DeviceState int

              DeviceState represents current devmapper device state reflected in meta store

              const (
              	// Unknown means that device just allocated and no operations were performed
              	Unknown DeviceState = iota
              	// Creating means that device is going to be created
              	// Created means that devices successfully created
              	// Activating means that device is going to be activated
              	// Activated means that device successfully activated
              	// Suspending means that device is going to be suspended
              	// Suspended means that device successfully suspended
              	// Resuming means that device is going to be resumed from suspended state
              	// Resumed means that device successfully resumed
              	// Deactivating means that device is going to be deactivated
              	// Deactivated means that device successfully deactivated
              	// Removing means that device is going to be removed
              	// Removed means that device successfully removed but not yet deleted from meta store
              	// Faulty means that the device is errored and the snapshotter failed to rollback it

              func (DeviceState) String

              func (s DeviceState) String() string

              type PoolDevice

              type PoolDevice struct {
              	// contains filtered or unexported fields

                PoolDevice ties together data and metadata volumes, represents thin-pool and manages volumes, snapshots and device ids.

                func NewPoolDevice

                func NewPoolDevice(ctx context.Context, config *Config) (*PoolDevice, error)

                  NewPoolDevice creates new thin-pool from existing data and metadata volumes. If pool 'poolName' already exists, it'll be reloaded with new parameters.

                  func (*PoolDevice) Close

                  func (p *PoolDevice) Close() error

                    Close closes pool device (thin-pool will not be removed)

                    func (*PoolDevice) CreateSnapshotDevice

                    func (p *PoolDevice) CreateSnapshotDevice(ctx context.Context, deviceName string, snapshotName string, virtualSizeBytes uint64) (retErr error)

                      CreateSnapshotDevice creates and activates new thin-device from parent thin-device (makes snapshot)

                      func (*PoolDevice) CreateThinDevice

                      func (p *PoolDevice) CreateThinDevice(ctx context.Context, deviceName string, virtualSizeBytes uint64) (retErr error)

                        CreateThinDevice creates new devmapper thin-device with given name and size. Device ID for thin-device will be allocated from metadata store. If allocation successful, device will be activated with /dev/mapper/<deviceName>

                        func (*PoolDevice) DeactivateDevice

                        func (p *PoolDevice) DeactivateDevice(ctx context.Context, deviceName string, deferred, withForce bool) error

                          DeactivateDevice deactivates thin device

                          func (*PoolDevice) GetUsage

                          func (p *PoolDevice) GetUsage(deviceName string) (int64, error)

                            GetUsage reports total size in bytes consumed by a thin-device. It relies on the number of used blocks reported by 'dmsetup status'. The output looks like:

                            device2: 0 204800 thin 17280 204799

                            Where 17280 is the number of used sectors

                            func (*PoolDevice) IsActivated

                            func (p *PoolDevice) IsActivated(deviceName string) bool

                              IsActivated returns true if thin-device is activated

                              func (*PoolDevice) IsLoaded

                              func (p *PoolDevice) IsLoaded(deviceName string) bool

                                IsLoaded returns true if thin-device is visible for dmsetup

                                func (*PoolDevice) MarkDeviceState

                                func (p *PoolDevice) MarkDeviceState(ctx context.Context, name string, state DeviceState) error

                                  MarkDeviceState changes the device's state in metastore

                                  func (*PoolDevice) RemoveDevice

                                  func (p *PoolDevice) RemoveDevice(ctx context.Context, deviceName string) error

                                    RemoveDevice completely wipes out thin device from thin-pool and frees it's device ID

                                    func (*PoolDevice) RemovePool

                                    func (p *PoolDevice) RemovePool(ctx context.Context) error

                                      RemovePool deactivates all child thin-devices and removes thin-pool device

                                      func (*PoolDevice) ResumeDevice

                                      func (p *PoolDevice) ResumeDevice(ctx context.Context, deviceName string) error

                                      func (*PoolDevice) SuspendDevice

                                      func (p *PoolDevice) SuspendDevice(ctx context.Context, deviceName string) error

                                        SuspendDevice flushes the outstanding IO and blocks the further IO

                                        func (*PoolDevice) WalkDevices

                                        func (p *PoolDevice) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

                                          WalkDevices iterates all devices in pool metadata

                                          type PoolMetadata

                                          type PoolMetadata struct {
                                          	// contains filtered or unexported fields

                                            PoolMetadata keeps device info for the given thin-pool device, it also responsible for generating next available device ids and tracking devmapper transaction numbers

                                            func NewPoolMetadata

                                            func NewPoolMetadata(dbfile string) (*PoolMetadata, error)

                                              NewPoolMetadata creates new or open existing pool metadata database

                                              func (*PoolMetadata) AddDevice

                                              func (m *PoolMetadata) AddDevice(ctx context.Context, info *DeviceInfo) error

                                                AddDevice saves device info to database.

                                                func (*PoolMetadata) ChangeDeviceState

                                                func (m *PoolMetadata) ChangeDeviceState(ctx context.Context, name string, state DeviceState) error

                                                  ChangeDeviceState changes the device state given the device name in devices bucket.

                                                  func (*PoolMetadata) Close

                                                  func (m *PoolMetadata) Close() error

                                                    Close closes metadata store

                                                    func (*PoolMetadata) GetDevice

                                                    func (m *PoolMetadata) GetDevice(ctx context.Context, name string) (*DeviceInfo, error)

                                                      GetDevice retrieves device info by name from database

                                                      func (*PoolMetadata) GetDeviceNames

                                                      func (m *PoolMetadata) GetDeviceNames(ctx context.Context) ([]string, error)

                                                        GetDeviceNames retrieves the list of device names currently stored in database

                                                        func (*PoolMetadata) MarkFaulty

                                                        func (m *PoolMetadata) MarkFaulty(ctx context.Context, name string) error

                                                          MarkFaulty marks the given device and corresponding devmapper device ID as faulty. The snapshotter might attempt to recreate a device in 'Faulty' state with another devmapper ID in subsequent calls, and in case of success it's status will be changed to 'Created' or 'Activated'. The devmapper dev ID will remain in 'deviceFaulty' state until manually handled by a user.

                                                          func (*PoolMetadata) RemoveDevice

                                                          func (m *PoolMetadata) RemoveDevice(ctx context.Context, name string) error

                                                            RemoveDevice removes device info from store.

                                                            func (*PoolMetadata) UpdateDevice

                                                            func (m *PoolMetadata) UpdateDevice(ctx context.Context, name string, fn DeviceInfoCallback) error

                                                              UpdateDevice updates device info in metadata store. The callback should be used to indicate whether device info update was successful or not. An error returned from the callback will rollback the update transaction in the database. Name and Device ID are not allowed to change.

                                                              func (*PoolMetadata) WalkDevices

                                                              func (m *PoolMetadata) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

                                                                WalkDevices walks all devmapper devices in metadata store and invokes the callback with device info. The provided callback function must not modify the bucket.

                                                                type Snapshotter

                                                                type Snapshotter struct {
                                                                	// contains filtered or unexported fields

                                                                  Snapshotter implements containerd's snapshotter ( based on Linux device-mapper targets.

                                                                  func NewSnapshotter

                                                                  func NewSnapshotter(ctx context.Context, config *Config) (*Snapshotter, error)

                                                                    NewSnapshotter creates new device mapper snapshotter. Internally it creates thin-pool device (or reloads if it's already exists) and initializes a database file for metadata.

                                                                    func (*Snapshotter) Cleanup

                                                                    func (s *Snapshotter) Cleanup(ctx context.Context) error

                                                                    func (*Snapshotter) Close

                                                                    func (s *Snapshotter) Close() error

                                                                      Close releases devmapper snapshotter resources. All subsequent Close calls will be ignored.

                                                                      func (*Snapshotter) Commit

                                                                      func (s *Snapshotter) Commit(ctx context.Context, name, key string, opts ...snapshots.Opt) error

                                                                        Commit marks an active snapshot as committed in meta store. Block device unmount operation captures snapshot changes by itself, so no additional actions needed within Commit operation.

                                                                        func (*Snapshotter) Mounts

                                                                        func (s *Snapshotter) Mounts(ctx context.Context, key string) ([]mount.Mount, error)

                                                                          Mounts return the list of mounts for the active or view snapshot

                                                                          func (*Snapshotter) Prepare

                                                                          func (s *Snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

                                                                            Prepare creates thin device for an active snapshot identified by key

                                                                            func (*Snapshotter) Remove

                                                                            func (s *Snapshotter) Remove(ctx context.Context, key string) error

                                                                              Remove removes thin device and snapshot metadata by key

                                                                              func (*Snapshotter) ResetPool

                                                                              func (s *Snapshotter) ResetPool(ctx context.Context) error

                                                                                ResetPool deactivates and deletes all thin devices in thin-pool. Used for cleaning pool after benchmarking.

                                                                                func (*Snapshotter) Stat

                                                                                func (s *Snapshotter) Stat(ctx context.Context, key string) (snapshots.Info, error)

                                                                                  Stat returns the info for an active or committed snapshot from store

                                                                                  func (*Snapshotter) Update

                                                                                  func (s *Snapshotter) Update(ctx context.Context, info snapshots.Info, fieldpaths ...string) (snapshots.Info, error)

                                                                                    Update updates an existing snapshot info's data

                                                                                    func (*Snapshotter) Usage

                                                                                    func (s *Snapshotter) Usage(ctx context.Context, key string) (snapshots.Usage, error)

                                                                                      Usage returns the resource usage of an active or committed snapshot excluding the usage of parent snapshots.

                                                                                      func (*Snapshotter) View

                                                                                      func (s *Snapshotter) View(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

                                                                                        View creates readonly thin device for the given snapshot key

                                                                                        func (*Snapshotter) Walk

                                                                                        func (s *Snapshotter) Walk(ctx context.Context, fn snapshots.WalkFunc, fs ...string) error

                                                                                          Walk iterates through all metadata Info for the stored snapshots and calls the provided function for each.


                                                                                          Path Synopsis