Version: v1.6.6 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jun 6, 2022 License: Apache-2.0 Imports: 24 Imported by: 42


Devmapper snapshotter

Devmapper is a containerd snapshotter plugin that stores snapshots in ext4-formatted filesystem images in a devicemapper thin pool.


To make it work you need to prepare thin-pool in advance and update containerd's configuration file. This file is typically located at /etc/containerd/config.toml.

Here's minimal sample entry that can be made in the configuration file:

    pool_name = "containerd-pool"
    base_image_size = "8192MB"

The following configuration flags are supported:

  • root_path - a directory where the metadata will be available (if empty default location for containerd plugins will be used)
  • pool_name - a name to use for the devicemapper thin pool. Pool name should be the same as in /dev/mapper/ directory
  • base_image_size - defines how much space to allocate when creating the base device
  • async_remove - flag to async remove device using snapshot GC's cleanup callback
  • discard_blocks - whether to discard blocks when removing a device. This is especially useful for returning disk space to the filesystem when using loopback devices.
  • fs_type - defines the file system to use for snapshot device mount. Valid values are ext4 and xfs. Defaults to ext4 if unspecified.
  • fs_options - optionally defines the file system options. This is currently only applicable to ext4 file system.

Pool name and base image size are required snapshotter parameters.


Give it a try with the following commands:

ctr images pull --snapshotter devmapper
ctr run --snapshotter devmapper test


The devicemapper snapshotter requires dmsetup (>= 1.02.110) command line tool to be installed and available on your computer. On Ubuntu, it can be installed with apt-get install dmsetup command.

How to setup device mapper thin-pool

There are many ways how to configure a devmapper thin-pool depending on your requirements, disk configuration, and environment.

On local dev environment you can utilize loopback devices. This type of configuration is simple and suits well for development and testing (please note that this configuration is slow and not recommended for production uses). Run the following script to create a thin-pool device:

set -ex


mkdir -p ${DATA_DIR}

# Create data file
sudo touch "${DATA_DIR}/data"
sudo truncate -s 100G "${DATA_DIR}/data"

# Create metadata file
sudo touch "${DATA_DIR}/meta"
sudo truncate -s 10G "${DATA_DIR}/meta"

# Allocate loop devices
DATA_DEV=$(sudo losetup --find --show "${DATA_DIR}/data")
META_DEV=$(sudo losetup --find --show "${DATA_DIR}/meta")

# Define thin-pool parameters.
# See for details.
DATA_SIZE="$(sudo blockdev --getsize64 -q ${DATA_DEV})"

# Create a thin-pool device
sudo dmsetup create "${POOL_NAME}" \

cat << EOF
# Add this to your config.toml configuration file and restart containerd daemon
    pool_name = "${POOL_NAME}"
    root_path = "${DATA_DIR}"
    base_image_size = "10GB"
    discard_blocks = true

Use dmsetup to verify that the thin-pool created successfully:

sudo dmsetup ls
devpool	(253:0)

Once configured and restarted containerd, you'll see the following output:

INFO[2020-03-17T20:24:45.532604888Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
INFO[2020-03-17T20:24:45.532672738Z] initializing pool device "dev-pool"

Another way to setup a thin-pool is via container-storage-setup tool (formerly known as docker-storage-setup). It is a script to configure CoW file systems like devicemapper:

set -ex

# Block device to use for devmapper thin-pool

# Install container-storage-setup tool
git clone
cd container-storage-setup/
sudo make install-core
echo "Using version $(container-storage-setup -v)"

# Create configuration file
# Refer to `man container-storage-setup` to see available options
sudo tee /etc/sysconfig/docker-storage-setup <<EOF

# Run the script
sudo container-storage-setup

cat << EOF
# Add this to your config.toml configuration file and restart containerd daemon
    pool_name = "${VG_NAME}-${POOL_NAME}"
    base_image_size = "10GB"

If successful container-storage-setup will output:

+ echo VG=containerd
+ sudo container-storage-setup
INFO: Volume group backing root filesystem could not be determined
INFO: Writing zeros to first 4MB of device /dev/xvdf
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 0.0162906 s, 257 MB/s
INFO: Device node /dev/xvdf1 exists.
  Physical volume "/dev/xvdf1" successfully created.
  Volume group "containerd" successfully created
  Rounding up size to full physical extent 12.00 MiB
  Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
  Logical volume "devpool" created.
  Logical volume containerd/devpool changed.

And dmsetup will produce the following output:

sudo dmsetup ls
containerd-devpool          (253:2)
containerd-devpool_tdata    (253:1)
containerd-devpool_tmeta    (253:0)




This section is empty.


View Source
var (
	// ErrNotFound represents an error returned when object not found in meta store
	ErrNotFound = errors.New("not found")
	// ErrAlreadyExists represents an error returned when object can't be duplicated in meta store
	ErrAlreadyExists = errors.New("object already exists")


This section is empty.


type Config

type Config struct {
	// Device snapshotter root directory for metadata
	RootPath string `toml:"root_path"`

	// Name for 'thin-pool' device to be used by snapshotter (without /dev/mapper/ prefix)
	PoolName string `toml:"pool_name"`

	// Defines how much space to allocate when creating base image for container
	BaseImageSize      string `toml:"base_image_size"`
	BaseImageSizeBytes uint64 `toml:"-"`

	// Flag to async remove device using Cleanup() callback in snapshots GC
	AsyncRemove bool `toml:"async_remove"`

	// Whether to discard blocks when removing a thin device.
	DiscardBlocks bool `toml:"discard_blocks"`

	// Defines file system to use for snapshout device mount. Defaults to "ext4"
	FileSystemType fsType `toml:"fs_type"`

	// Defines optional file system options passed through config file
	FsOptions string `toml:"fs_options"`

Config represents device mapper configuration loaded from file. Size units can be specified in human-readable string format (like "32KIB", "32GB", "32Tb")

func LoadConfig

func LoadConfig(path string) (*Config, error)

LoadConfig reads devmapper configuration file from disk in TOML format

func (*Config) Validate

func (c *Config) Validate() error

Validate makes sure configuration fields are valid

type DeviceInfo

type DeviceInfo struct {
	// DeviceID is a 24-bit number assigned to a device within thin-pool device
	DeviceID uint32 `json:"device_id"`
	// Size is a thin device size
	Size uint64 `json:"size"`
	// Name is a device name to be used in /dev/mapper/
	Name string `json:"name"`
	// ParentName is a name of parent device (if snapshot)
	ParentName string `json:"parent_name"`
	// State represents current device state
	State DeviceState `json:"state"`
	// Error details if device state change failed
	Error string `json:"error"`

DeviceInfo represents metadata for thin device within thin-pool

type DeviceInfoCallback

type DeviceInfoCallback func(deviceInfo *DeviceInfo) error

DeviceInfoCallback is a callback used for device updates

type DeviceState

type DeviceState int

DeviceState represents current devmapper device state reflected in meta store

const (
	// Unknown means that device just allocated and no operations were performed
	Unknown DeviceState = iota
	// Creating means that device is going to be created
	// Created means that devices successfully created
	// Activating means that device is going to be activated
	// Activated means that device successfully activated
	// Suspending means that device is going to be suspended
	// Suspended means that device successfully suspended
	// Resuming means that device is going to be resumed from suspended state
	// Resumed means that device successfully resumed
	// Deactivating means that device is going to be deactivated
	// Deactivated means that device successfully deactivated
	// Removing means that device is going to be removed
	// Removed means that device successfully removed but not yet deleted from meta store
	// Faulty means that the device is errored and the snapshotter failed to rollback it

func (DeviceState) String

func (s DeviceState) String() string

type PoolDevice

type PoolDevice struct {
	// contains filtered or unexported fields

PoolDevice ties together data and metadata volumes, represents thin-pool and manages volumes, snapshots and device ids.

func NewPoolDevice

func NewPoolDevice(ctx context.Context, config *Config) (*PoolDevice, error)

NewPoolDevice creates new thin-pool from existing data and metadata volumes. If pool 'poolName' already exists, it'll be reloaded with new parameters.

func (*PoolDevice) Close

func (p *PoolDevice) Close() error

Close closes pool device (thin-pool will not be removed)

func (*PoolDevice) CreateSnapshotDevice

func (p *PoolDevice) CreateSnapshotDevice(ctx context.Context, deviceName string, snapshotName string, virtualSizeBytes uint64) (retErr error)

CreateSnapshotDevice creates and activates new thin-device from parent thin-device (makes snapshot)

func (*PoolDevice) CreateThinDevice

func (p *PoolDevice) CreateThinDevice(ctx context.Context, deviceName string, virtualSizeBytes uint64) (retErr error)

CreateThinDevice creates new devmapper thin-device with given name and size. Device ID for thin-device will be allocated from metadata store. If allocation successful, device will be activated with /dev/mapper/<deviceName>

func (*PoolDevice) DeactivateDevice

func (p *PoolDevice) DeactivateDevice(ctx context.Context, deviceName string, deferred, withForce bool) error

DeactivateDevice deactivates thin device

func (*PoolDevice) GetUsage

func (p *PoolDevice) GetUsage(deviceName string) (int64, error)

GetUsage reports total size in bytes consumed by a thin-device. It relies on the number of used blocks reported by 'dmsetup status'. The output looks like:

device2: 0 204800 thin 17280 204799

Where 17280 is the number of used sectors

func (*PoolDevice) IsActivated

func (p *PoolDevice) IsActivated(deviceName string) bool

IsActivated returns true if thin-device is activated

func (*PoolDevice) IsLoaded

func (p *PoolDevice) IsLoaded(deviceName string) bool

IsLoaded returns true if thin-device is visible for dmsetup

func (*PoolDevice) MarkDeviceState added in v1.4.0

func (p *PoolDevice) MarkDeviceState(ctx context.Context, name string, state DeviceState) error

MarkDeviceState changes the device's state in metastore

func (*PoolDevice) RemoveDevice

func (p *PoolDevice) RemoveDevice(ctx context.Context, deviceName string) error

RemoveDevice completely wipes out thin device from thin-pool and frees it's device ID

func (*PoolDevice) RemovePool

func (p *PoolDevice) RemovePool(ctx context.Context) error

RemovePool deactivates all child thin-devices and removes thin-pool device

func (*PoolDevice) ResumeDevice added in v1.3.7

func (p *PoolDevice) ResumeDevice(ctx context.Context, deviceName string) error

ResumeDevice resumes IO for the given device

func (*PoolDevice) SuspendDevice

func (p *PoolDevice) SuspendDevice(ctx context.Context, deviceName string) error

SuspendDevice flushes the outstanding IO and blocks the further IO

func (*PoolDevice) WalkDevices added in v1.4.0

func (p *PoolDevice) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

WalkDevices iterates all devices in pool metadata

type PoolMetadata

type PoolMetadata struct {
	// contains filtered or unexported fields

PoolMetadata keeps device info for the given thin-pool device, it also responsible for generating next available device ids and tracking devmapper transaction numbers

func NewPoolMetadata

func NewPoolMetadata(dbfile string) (*PoolMetadata, error)

NewPoolMetadata creates new or open existing pool metadata database

func (*PoolMetadata) AddDevice

func (m *PoolMetadata) AddDevice(ctx context.Context, info *DeviceInfo) error

AddDevice saves device info to database.

func (*PoolMetadata) ChangeDeviceState added in v1.4.0

func (m *PoolMetadata) ChangeDeviceState(ctx context.Context, name string, state DeviceState) error

ChangeDeviceState changes the device state given the device name in devices bucket.

func (*PoolMetadata) Close

func (m *PoolMetadata) Close() error

Close closes metadata store

func (*PoolMetadata) GetDevice

func (m *PoolMetadata) GetDevice(ctx context.Context, name string) (*DeviceInfo, error)

GetDevice retrieves device info by name from database

func (*PoolMetadata) GetDeviceNames

func (m *PoolMetadata) GetDeviceNames(ctx context.Context) ([]string, error)

GetDeviceNames retrieves the list of device names currently stored in database

func (*PoolMetadata) MarkFaulty

func (m *PoolMetadata) MarkFaulty(ctx context.Context, name string) error

MarkFaulty marks the given device and corresponding devmapper device ID as faulty. The snapshotter might attempt to recreate a device in 'Faulty' state with another devmapper ID in subsequent calls, and in case of success it's status will be changed to 'Created' or 'Activated'. The devmapper dev ID will remain in 'deviceFaulty' state until manually handled by a user.

func (*PoolMetadata) RemoveDevice

func (m *PoolMetadata) RemoveDevice(ctx context.Context, name string) error

RemoveDevice removes device info from store.

func (*PoolMetadata) UpdateDevice

func (m *PoolMetadata) UpdateDevice(ctx context.Context, name string, fn DeviceInfoCallback) error

UpdateDevice updates device info in metadata store. The callback should be used to indicate whether device info update was successful or not. An error returned from the callback will rollback the update transaction in the database. Name and Device ID are not allowed to change.

func (*PoolMetadata) WalkDevices

func (m *PoolMetadata) WalkDevices(ctx context.Context, cb func(info *DeviceInfo) error) error

WalkDevices walks all devmapper devices in metadata store and invokes the callback with device info. The provided callback function must not modify the bucket.

type Snapshotter

type Snapshotter struct {
	// contains filtered or unexported fields

Snapshotter implements containerd's snapshotter ( based on Linux device-mapper targets.

func NewSnapshotter

func NewSnapshotter(ctx context.Context, config *Config) (*Snapshotter, error)

NewSnapshotter creates new device mapper snapshotter. Internally it creates thin-pool device (or reloads if it's already exists) and initializes a database file for metadata.

func (*Snapshotter) Cleanup added in v1.4.0

func (s *Snapshotter) Cleanup(ctx context.Context) error

Cleanup cleans up all removed and unused resources

func (*Snapshotter) Close

func (s *Snapshotter) Close() error

Close releases devmapper snapshotter resources. All subsequent Close calls will be ignored.

func (*Snapshotter) Commit

func (s *Snapshotter) Commit(ctx context.Context, name, key string, opts ...snapshots.Opt) error

Commit marks an active snapshot as committed in meta store. Block device unmount operation captures snapshot changes by itself, so no additional actions needed within Commit operation.

func (*Snapshotter) Mounts

func (s *Snapshotter) Mounts(ctx context.Context, key string) ([]mount.Mount, error)

Mounts return the list of mounts for the active or view snapshot

func (*Snapshotter) Prepare

func (s *Snapshotter) Prepare(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

Prepare creates thin device for an active snapshot identified by key

func (*Snapshotter) Remove

func (s *Snapshotter) Remove(ctx context.Context, key string) error

Remove removes thin device and snapshot metadata by key

func (*Snapshotter) ResetPool

func (s *Snapshotter) ResetPool(ctx context.Context) error

ResetPool deactivates and deletes all thin devices in thin-pool. Used for cleaning pool after benchmarking.

func (*Snapshotter) Stat

func (s *Snapshotter) Stat(ctx context.Context, key string) (snapshots.Info, error)

Stat returns the info for an active or committed snapshot from store

func (*Snapshotter) Update

func (s *Snapshotter) Update(ctx context.Context, info snapshots.Info, fieldpaths ...string) (snapshots.Info, error)

Update updates an existing snapshot info's data

func (*Snapshotter) Usage

func (s *Snapshotter) Usage(ctx context.Context, key string) (snapshots.Usage, error)

Usage returns the resource usage of an active or committed snapshot excluding the usage of parent snapshots.

func (*Snapshotter) View

func (s *Snapshotter) View(ctx context.Context, key, parent string, opts ...snapshots.Opt) ([]mount.Mount, error)

View creates readonly thin device for the given snapshot key

func (*Snapshotter) Walk

func (s *Snapshotter) Walk(ctx context.Context, fn snapshots.WalkFunc, fs ...string) error

Walk iterates through all metadata Info for the stored snapshots and calls the provided function for each.


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL