ark

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

README

ark

ark

Deduplicated backups as a Go library, with a CLI on top.

ark backs up anything you can walk as a tree of files: a directory, a Docker volume, a database dump, an API export. Content is split with FastCDC, stored once, compressed with zstd and encrypted.

  • github.com/alexjoedt/ark is the library module: the core package ark, the backends arklocal, arksftp and arkmem, arkuri, arkfs for directories, the test suites in arktest and chunker.
  • github.com/alexjoedt/ark/arkdocker adds Docker volumes, in its own module so the Docker SDK stays out of the core.
  • github.com/alexjoedt/ark/cmd/ark is the ark binary, a restic-style CLI with an optional web UI.

Library

go get github.com/alexjoedt/ark

Back up a directory and restore it:

b, err := arklocal.New("/srv/ark")
if err != nil {
	return err
}
pw := ark.WithPassword(os.Getenv("ARK_PASSWORD"))
if err := ark.Init(ctx, b, pw); err != nil { // once per repository
	return err
}
repo, err := ark.Open(ctx, b, pw)
if err != nil {
	return err
}
defer repo.Close()

snap, err := arkfs.Backup(ctx, repo, "/home/me/docs", ark.WithTags("daily"))
if err != nil {
	return err
}
_, err = arkfs.Restore(ctx, repo, snap.ID, "/tmp/docs")

Open never creates a repository, so a mistyped path fails with ark.ErrNotInitialized instead of starting a second one. Restore, export and verify stream the snapshot: memory stays at the chunk cache (256 MiB by default) no matter how large the snapshot is.

Packages
Package What it is
Core
ark Repositories, snapshots, backup, restore, forget, prune, verify, check, export, import, locks
chunker FastCDC content-defined chunking
Backends, where the repository lives
arklocal A local directory, atomic writes through temp file and rename
arksftp A directory over SFTP, host key checked against known_hosts by default
arkmem In memory, for tests
arkuri Opens one of the above from a string
Sources and sinks, what gets backed up and restored
arkfs A local directory, with hardlinks and xattrs
arkdocker A Docker volume, and an Engine that stops and restarts the containers using it
Testing
arktest Conformance suites for your own backends, sources and sinks
Backends

A backend stores opaque, encrypted objects. Pick one and hand it to ark.Init and ark.Open:

b, err := arklocal.New("/srv/ark")

b, err := arksftp.New(arksftp.Config{Host: "nas", User: "backup", Path: "/srv/ark"})

b, err := arkuri.Open(ctx, "sftp://backup@nas/srv/ark") // or "/srv/ark", "local:///srv/ark"

b := arkmem.New() // its objects are gone on Close

arksftp reads ~/.ssh/config, so a Host alias works. It authenticates with ssh-agent, then IdentityFile or the default keys in ~/.ssh, then the password in ARK_SFTP_PASSWORD. An encrypted key file is unlocked with ARK_SFTP_KEY_PASSPHRASE. Secrets never go into Config or a URI. arkuri takes the non-secret options as query parameters: known_hosts, identity and insecure.

Anything else, such as S3 or a NAS API, is an ark.Backend: seven methods (Put, Get, Exists, Delete, List, Stat, Close). The suite in arktest checks the same contract the built-in backends pass:

func TestConformance(t *testing.T) {
	arktest.RunBackendTests(t, func(t *testing.T) ark.Backend {
		return mybackend.New()
	})
}
Sources and sinks

A backup reads an ark.Source and a restore writes into an ark.Sink. arkfs.Backup and arkfs.Restore above wrap repo.Backup(ctx, src) and repo.Restore(ctx, id, sink) for a directory. A Docker volume goes through arkdocker, which stops the containers using the volume and starts them again:

eng, err := arkdocker.New(repo) // DOCKER_HOST, or arkdocker.WithDockerHost
snap, err := eng.BackupVolume(ctx, "postgres_data")
_, err = eng.RestoreVolume(ctx, snap.ID, "postgres_copy") // created if missing

Your own data goes in through ark.Source, two methods: Meta names the snapshot, Walk yields entries. A matching ark.Sink (Open, Receive, Close) brings it back, and arktest.RunSourceSinkTests checks that what the sink receives, the source yields again.

examples/ has three runnable programs:

Example Shows
examples/filebackup a directory into a local repository and back
examples/dockervolume a Docker volume into a snapshot and into a second volume
examples/customsource an ark.Source over in-memory files
Encryption

Every repository is encrypted and authenticated. The password unlocks a random master key stored in keys/; chunks, trees, snapshot manifests and locks are sealed with XChaCha20-Poly1305 and named by an HMAC, so the backend sees neither contents nor file names. A changed object fails to open instead of restoring garbage.

  • A lost password is lost data. There is no recovery and no backdoor.
  • ark.WithoutPassword() creates or opens a repository without one. The format is the same, but the master key is stored unprotected: anyone with access to the backend can read it.
  • A wrong password fails with ark.ErrWrongPassword before any data is read.
  • repo.ChangePassword seals the same master key under a new password. It does not revoke a key that already leaked.
  • Open derives the key with argon2id and needs 64 MiB of memory for it. Size containers accordingly.

CLI

go install github.com/alexjoedt/ark/cmd/ark@latest

Every command needs a repository through --repo or ARK_REPOSITORY. There is no default path. The password comes from, in this order:

Source Flag or variable
a file, one trailing newline stripped --password-file or ARK_PASSWORD_FILE
the environment ARK_PASSWORD
a prompt, only on a terminal ark init asks twice
no password at all --insecure-no-password or ARK_INSECURE_NO_PASSWORD=1

There is no --password flag, it would show in the process list. ark key passwd changes the password and reads the new one from --new-password-file, ARK_NEW_PASSWORD or a prompt; --new-insecure-no-password removes it.

export ARK_REPOSITORY=/srv/ark        # or sftp://backup@nas/srv/ark
export ARK_PASSWORD_FILE=/etc/ark/password

ark init
ark backup /etc --tag daily
ark snapshots
ark ls <id>
ark restore <id> /tmp/etc-restore
ark forget --keep-daily 7 --keep-weekly 4 --prune
ark check

Docker volumes live under ark docker. The daemon comes from --docker-host or DOCKER_HOST.

ark docker volumes
ark docker backup postgres_data --comment "before upgrade"
ark snapshots --type docker-volume    # only volume snapshots, file backups have type file
ark docker restore <id>                # into the volume it came from
ark docker restore <id> postgres_copy  # or into another one, created if missing

ark restore refuses Docker volume snapshots. Use ark docker restore, or unpack one into a directory with ark export <id> | tar x.

Destructive commands ask for confirmation on a terminal. Without one they fail unless --yes is set. --json prints results on stdout and logs on stderr. ark <command> --help lists every flag.

Exit code Meaning
0 success
1 any other failure
2 usage error
3 repository locked
4 snapshot not found, or an id prefix matches several
5 verify or check found damage
6 wrong password

Compatibility

From 1.0.0 on, ark follows semantic versioning. All three modules release together under one version number.

  • The exported API of ark, arkfs, arklocal, arksftp, arkuri, arkmem, arktest, chunker and arkdocker follows the Go compatibility rules: minors add, only a major removes.
  • Every 1.x release reads the v1 on-disk format. Minors may add optional fields.
  • CLI flags keep their meaning. --json output only gains fields.
  • ark export writes plain tar, and ark import reads it.
  • Not covered: dependency versions (including the Docker SDK), the web UI, and ark serve.

Web UI

ark serve starts a web UI on :8080 with scheduled backups and live logs. It is not covered by the compatibility promise.

ARK_ADMIN_PASSWORD=... ARK_PASSWORD_FILE=/etc/ark/password ark serve --repo /srv/ark

ark serve has no terminal to prompt on, so the repository password comes from a file or the environment, ARK_PASSWORD or ARK_PASSWORD_FILE, and the repository is created once with ark init before ark serve starts.

The SQLite database next to it is a cache for the UI. The repository stays the source of truth.

scripts/install.sh installs ark serve as a systemd service with its settings in /etc/ark/daemon.json. The admin password only ever comes from ARK_ADMIN_PASSWORD, never from a file.

Repository layout

/srv/ark/
├── config      format version and chunker settings, plaintext
├── keys/       the master key, sealed under the password
├── chunks/     compressed, encrypted content, named by an HMAC, sharded by prefix
├── trees/      one encrypted object per directory, named the same way
├── snapshots/  one encrypted manifest per snapshot, named by its sortable id
└── locks/      encrypted shared and exclusive locks, stale after 30 minutes without a refresh

Development

Four modules in one workspace: the root, arkdocker/, cmd/ark/ and the untagged examples/.

task test    # go test across the workspace, then go mod verify
task vet
task lint    # golangci-lint in each tagged module
task build   # builds the web UI, then bin/ark
task tidy    # tidies every module and syncs the workspace

go build ./... stops at the root module. Use go build github.com/alexjoedt/ark/....

Releases: docs/RELEASING.md.

License

Apache License 2.0, see LICENSE.

Documentation

Overview

Package ark provides a general-purpose CDC (content-defined chunking) backup library. It offers restic-style deduplicating backups through a clean Source/Sink abstraction that any caller can implement.

A repository is created once with Init and opened with Open. Open never creates one, so a mistyped path fails instead of starting a second repository.

Basic usage:

b, _ := arklocal.New("/path/to/repo")
ark.Init(ctx, b, ark.WithPassword(pw)) // once, per repository

repo, _ := ark.Open(ctx, b, ark.WithPassword(pw))
snapshot, _ := repo.Backup(ctx, mySource)
repo.Restore(ctx, snapshot.ID, mySink)

Passwords

Every repository is encrypted and authenticated. Init creates a random master key and stores it in a key file sealed under the password; the data is encrypted with the master key, never with the password. Open unseals the master key before it reads any other object, and a wrong password fails with ErrWrongPassword. There is no recovery: a lost password is lost data.

A repository without a password is created and opened with WithoutPassword. It has the same format with the master key stored unprotected, so anyone with access to the backend can read and forge it; corruption is still detected. The choice is explicit on every Init and Open: WithoutPassword on a protected repository fails with ErrPasswordRequired, a password on an unprotected one with ErrNotPasswordProtected.

ChangePassword seals the same master key under a new password. It does not revoke a master key that leaked, through the old password or a copy of the old key file; a leaked repository needs a new one.

Init, Open and ChangePassword derive the key with argon2id at 64 MiB of memory, so a container that opens a repository needs that much on top of the restore cache.

Memory

Restore, Export and Verify stream: they walk the snapshot in order and read the chunks of the file at hand, so their memory is the restore chunk cache (256 MiB by default, see WithRestoreCache) plus one chunk per worker in flight, where the workers are capped by WithConcurrency. Neither the size of the snapshot nor the size of a single file changes that.

Index

Constants

View Source
const UnknownNamespace = "unknown"

UnknownNamespace is the namespace name String returns for a handle type this version of ark does not define. Backends compare against it to reject one.

Variables

View Source
var (
	// ErrNotFound is returned when a requested snapshot does not exist.
	ErrNotFound = errors.New("ark: snapshot not found")

	// ErrAmbiguousID is returned when a snapshot id prefix matches more than
	// one snapshot. The message names the candidates.
	ErrAmbiguousID = errors.New("ark: snapshot id prefix is ambiguous")

	// ErrEmptyPolicy is returned by Forget when the policy keeps nothing and
	// the filter names no ids, which would remove every matching snapshot.
	ErrEmptyPolicy = errors.New("ark: forget policy keeps nothing and no ids given")

	// ErrIncompatibleSourceType is returned when restoring a snapshot into a
	// Sink type that is incompatible with the snapshot's SourceType.
	ErrIncompatibleSourceType = errors.New("ark: incompatible source type")

	// ErrCorruptManifest is returned when a snapshot manifest does not
	// authenticate: it was changed, truncated or moved from another snapshot.
	// Snapshots, Prune and a Forget policy fail while one exists; remove it
	// by naming its id to Forget without a policy, then Prune.
	ErrCorruptManifest = errors.New("ark: corrupt snapshot manifest")

	// ErrCorruptTree is returned by Verify when a tree object of the snapshot
	// does not authenticate or does not match its name.
	ErrCorruptTree = errors.New("ark: corrupt snapshot tree")

	// ErrCorruptChunk is returned when a stored chunk fails its integrity check.
	ErrCorruptChunk = errors.New("ark: corrupt chunk")

	// ErrMissingChunk is returned when a required chunk is absent from the CAS.
	ErrMissingChunk = errors.New("ark: missing chunk")

	// ErrRepoLocked is returned when a repo-level lock cannot be acquired.
	ErrRepoLocked = errors.New("ark: repository is locked by another operation")

	// ErrNotInitialized is returned by Open when the backend holds no
	// repository config. Call Init first.
	ErrNotInitialized = errors.New("ark: repository is not initialized")

	// ErrAlreadyInitialized is returned by Init when the backend already
	// holds a repository config.
	ErrAlreadyInitialized = errors.New("ark: repository is already initialized")

	// ErrUnsupportedFormat is returned by Open when the repository config
	// names a format this build does not read.
	ErrUnsupportedFormat = errors.New("ark: unsupported repository format")

	// ErrWrongPassword is returned by Open when no key file of the
	// repository opens with the given password.
	ErrWrongPassword = errors.New("ark: wrong password or no key file matches")

	// ErrPasswordRequired is returned by Open when WithoutPassword is given
	// for a password protected repository.
	ErrPasswordRequired = errors.New("ark: repository is password protected")

	// ErrNotPasswordProtected is returned by Open when a password is given
	// for a repository created with WithoutPassword.
	ErrNotPasswordProtected = errors.New("ark: repository is not password protected")
)

Sentinel errors returned by Repository methods.

Functions

func Init

func Init(ctx context.Context, b Backend, pw Passphrase, opts ...InitOption) error

Init creates a repository in b: a fresh master key sealed under pw in a key file, then the config object. pw is WithPassword or, explicitly, WithoutPassword. The password only unlocks the master key; without it the repository cannot be read, and nothing recovers it.

It returns ErrAlreadyInitialized if b already holds a config, so a typo in a repository path cannot silently create a second repository next to the first. The config is written last, so a failed Init leaves no repository behind.

Types

type Backend

type Backend interface {
	// Put stores r under h atomically: a reader never observes a partial
	// object. Implementations write to a temporary name in the same
	// namespace and rename it into place. Put is idempotent, an existing
	// object is replaced. Put need not make the object durable; Sync does.
	Put(ctx context.Context, h Handle, r io.Reader) error

	// Get retrieves a reader for the object stored under h.
	// Callers must close the returned ReadCloser.
	Get(ctx context.Context, h Handle) (io.ReadCloser, error)

	// Exists returns true if the object identified by h is present.
	Exists(ctx context.Context, h Handle) (bool, error)

	// Delete removes the object identified by h.
	// Removing a non-existent object is not an error.
	Delete(ctx context.Context, h Handle) error

	// Sync is the durability barrier: every Put and Delete that returned
	// before the call has reached stable storage when Sync returns, so a
	// power loss cannot undo it. A backend that already makes each Put and
	// Delete durable returns nil without doing anything.
	Sync(ctx context.Context) error

	// List calls fn once for each object of the given type with its name and
	// stored size. If fn returns an error, List stops and returns it.
	List(ctx context.Context, t HandleType, fn func(BlobInfo) error) error

	// Stat returns metadata for the object stored under h.
	Stat(ctx context.Context, h Handle) (BlobInfo, error)

	// Close releases any resources held by the backend.
	Close() error
}

Backend is the storage abstraction for all repository objects.

All methods must be safe for concurrent use. The core issues at most WithConcurrency operations at a time, defaulting to runtime.NumCPU*2, so a remote backend can size its connection pool from that bound.

A backend maps Handle.Type to a namespace by its String() value and applies sharding only to BlobFile. An unknown type maps to the namespace "unknown"; implementations may reject it instead.

type BackupOption

type BackupOption func(*backupOptions)

BackupOption is a functional option for Backup.

func WithBackupProgress

func WithBackupProgress(fn ProgressFunc) BackupOption

WithBackupProgress sets the observer for this backup. One observer per call, so two operations on the same Repository never interleave their events. fn must not block.

func WithComment

func WithComment(s string) BackupOption

WithComment attaches a human-readable description to the snapshot.

func WithHost

func WithHost(h string) BackupOption

WithHost overrides the hostname recorded in the snapshot.

func WithParent

func WithParent(id string) BackupOption

WithParent names the parent snapshot explicitly, as a full id or a unique prefix. Without it the parent is the newest snapshot of the same source. A parent of another source is recorded but no file is taken from it unread.

func WithTags

func WithTags(tags ...string) BackupOption

WithTags attaches one or more tags to the snapshot.

type BlobInfo

type BlobInfo struct {
	Name string
	Size int64 // size in bytes as stored (e.g. compressed)
}

BlobInfo describes a stored object.

type CheckResult

type CheckResult struct {
	SnapshotsChecked int
	Errors           []string
	// Warnings are findings that leave every snapshot intact, such as a
	// backup that did not finish; Prune clears them.
	Warnings []string
}

CheckResult is returned by Check, which walks all snapshots.

type ConcurrentSource added in v0.2.0

type ConcurrentSource interface {
	Source
	ConcurrentOpen() bool
}

ConcurrentSource is a Source whose open functions outlive the WalkFunc call that passed them: open may run after fn returned, on any goroutine and concurrently with the walk and with other opens. A backup reads the files of such a source in parallel; it reads any other source inside fn, one file at a time. ConcurrentOpen returning false opts a wrapper out.

type FileInfo

type FileInfo struct {
	Path     string            `json:"path"`
	Size     int64             `json:"size"`
	Mode     os.FileMode       `json:"mode"`
	UID      int               `json:"uid"`
	GID      int               `json:"gid"`
	ModTime  time.Time         `json:"mtime"`
	Type     FileType          `json:"type"`
	LinkDest string            `json:"link_dest,omitempty"`
	Xattrs   map[string][]byte `json:"xattrs,omitempty"`

	// ChangeTime and Inode let a backup trust that a regular file is
	// unchanged since its parent snapshot: ctime cannot be set from user
	// space, so it catches a rewrite that kept the mtime. A source that
	// cannot tell (a tar stream) leaves both zero, and only Size and ModTime
	// are compared.
	ChangeTime time.Time `json:"ctime,omitzero"`
	Inode      uint64    `json:"inode,omitempty"`
}

FileInfo describes a single entry yielded by a Source or received by a Sink.

The JSON tags are a convenience for callers that log or print entries. They are not the repository wire format; the tree entry type owns that.

type FileType

type FileType string

FileType represents the type of a filesystem entry.

const (
	// FileTypeRegular is a normal file.
	FileTypeRegular FileType = "regular"

	// FileTypeDirectory is a directory.
	FileTypeDirectory FileType = "directory"

	// FileTypeSymlink is a symbolic link.
	FileTypeSymlink FileType = "symlink"

	// FileTypeHardlink is a second name for a file already yielded under
	// another path. LinkDest names that earlier path and the content is
	// stored once, under the first name.
	FileTypeHardlink FileType = "hardlink"

	// FileTypeUnsupported marks an entry the v1 format does not model:
	// device nodes, fifos, sockets. The core drops it and counts it in
	// SnapshotStats.Skipped.
	FileTypeUnsupported FileType = "unsupported"
)

type ForgetOption

type ForgetOption func(*forgetConfig)

ForgetOption configures a Forget call.

func WithForgetDryRun

func WithForgetDryRun() ForgetOption

WithForgetDryRun makes Forget report what would be removed without deleting anything.

type ForgetPolicy

type ForgetPolicy struct {
	// KeepLast keeps the most recent n snapshots regardless of age.
	KeepLast int

	// KeepHourly keeps one snapshot per hour for the most recent n hours.
	KeepHourly int

	// KeepDaily keeps one snapshot per day for the most recent n days.
	KeepDaily int

	// KeepWeekly keeps one snapshot per ISO week for the most recent n weeks.
	KeepWeekly int

	// KeepMonthly keeps one snapshot per month for the most recent n months.
	KeepMonthly int

	// KeepYearly keeps one snapshot per year for the most recent n years.
	KeepYearly int

	// KeepWithin keeps every snapshot younger than this, measured from the
	// newest snapshot of the group rather than from the wall clock, so a
	// repository that stopped receiving backups does not lose everything.
	KeepWithin time.Duration
}

ForgetPolicy defines the retention rules applied by Repository.Forget. The policy is applied per source name, so a busy source never pushes another source's snapshots out of the window. Zero value keeps everything.

type ForgetResult

type ForgetResult struct {
	Keep   []*Snapshot
	Remove []*Snapshot
	DryRun bool
}

ForgetResult is returned by Repository.Forget.

type Handle

type Handle struct {
	Type HandleType
	Name string
}

Handle identifies a single object in the Backend.

type HandleType

type HandleType uint8

HandleType identifies the category of an object stored in a Backend. Each type owns a namespace named by String(); a backend maps that name to a directory or key prefix. New types may appear in minor versions, so a backend must derive its layout from String() instead of switching on the constants.

const (
	// ConfigFile is the repository config object. One per repository.
	ConfigFile HandleType = 1 + iota
	// BlobFile is a CAS content chunk. Backends shard these by name prefix.
	BlobFile
	// TreeFile is a tree node, named by the hash of its canonical JSON.
	TreeFile
	// SnapshotFile is a snapshot manifest, named by the snapshot id.
	SnapshotFile
	// LockFile is a repository lock object.
	LockFile
	// KeyFile holds the master key sealed under one password.
	KeyFile
	// MarkerFile marks a backup in progress, named like its lock. One left
	// behind by a backup that never finished makes later backups distrust
	// Exists until Prune.
	MarkerFile
)

func (HandleType) String

func (t HandleType) String() string

String returns the namespace name of the handle type, or UnknownNamespace for a type this version of ark does not define.

type InitOption

type InitOption func(*initConfig)

InitOption configures the repository Init writes. Options are frozen into the config object, so they apply to every later Open of that repository.

func WithChunkerConfig

func WithChunkerConfig(c chunker.Config) InitOption

WithChunkerConfig stores custom FastCDC boundaries in the repository config. Zero fields fall back to the chunker defaults. Seed and IDKey are ignored, the repository's master key supplies them. The parameters cannot be changed afterwards without breaking deduplication against existing chunks.

type LockInfo

type LockInfo struct {
	ID        string
	Time      time.Time
	Exclusive bool
	Hostname  string
	Username  string
	PID       int
	Stale     bool
}

LockInfo describes one lock object in the repository.

type Option

type Option func(*openConfig)

Option configures a Repository at Open time.

func WithConcurrency

func WithConcurrency(n int) Option

WithConcurrency caps how many backend operations run at once. Values below one fall back to the default, twice the number of CPUs.

func WithLockWait

func WithLockWait(d time.Duration) Option

WithLockWait makes an operation retry a blocked lock for up to d instead of failing at once with ErrRepoLocked. The default is zero, no waiting.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger for the repository.

type Passphrase

type Passphrase struct {
	// contains filtered or unexported fields
}

Passphrase is the password of a repository, or the explicit statement that it has none. WithPassword and WithoutPassword are its only constructors; the zero value is rejected by Init and Open. The package documentation, section Passwords, describes what the password protects and what it does not.

func WithPassword

func WithPassword(password string) Passphrase

WithPassword returns the passphrase for a password protected repository. An empty password is rejected by Init and Open; use WithoutPassword.

func WithoutPassword

func WithoutPassword() Passphrase

WithoutPassword returns the passphrase for a repository without a password. Its master key is stored unprotected: anyone with access to the backend can read and forge its contents, only corruption is still detected.

func (Passphrase) GoString

func (p Passphrase) GoString() string

GoString keeps %#v from printing the password.

func (Passphrase) String

func (p Passphrase) String() string

String never reveals the password.

type Phase

type Phase string

Phase names the step an operation is currently in. The values are stable identifiers, safe to compare and to emit as JSON; rendering them for humans is the caller's job.

const (
	// PhaseScan opens a backup, PhaseStore and PhaseFinalize follow.
	PhaseScan     Phase = "scan"     // walking the source
	PhaseStore    Phase = "store"    // chunking and writing
	PhaseFinalize Phase = "finalize" // tree and manifest

	// PhaseLoad opens a restore, PhaseFetch follows.
	PhaseLoad  Phase = "load"  // manifest and tree
	PhaseFetch Phase = "fetch" // chunks
	PhaseWrite Phase = "write" // sink
)

type ProgressEvent

type ProgressEvent struct {
	Phase Phase

	// CurrentPath is the entry the event reports, empty when the phase works
	// on no single entry. A backup reports a file once it is stored; files
	// read in parallel finish, and are reported, out of walk order.
	CurrentPath string

	FilesTotal  int
	FilesDone   int
	BytesTotal  int64
	BytesDone   int64
	ChunksTotal int
	ChunksDone  int
}

ProgressEvent describes the current progress of a backup or restore.

type ProgressFunc

type ProgressFunc func(ProgressEvent)

ProgressFunc receives progress events from one backup or restore operation. It must not block; implementations should drop events when the consumer is slow. Calls never overlap, so a handler may update its state without a lock. The phases of a restore interleave: chunks of upcoming files are fetched while the current file is written.

Example with a channel:

ch := make(chan ark.ProgressEvent, 32)
snap, err := repo.Backup(ctx, src, ark.WithBackupProgress(func(e ark.ProgressEvent) {
	select { case ch <- e: default: }
}))

type PruneOption

type PruneOption func(*pruneConfig)

PruneOption configures a Prune call.

func WithPruneDryRun

func WithPruneDryRun() PruneOption

WithPruneDryRun makes Prune report what would be removed without deleting anything.

type PruneResult

type PruneResult struct {
	BlobsRemoved int
	BytesFreed   int64
	DryRun       bool
}

PruneResult is returned by Prune.

type Repository

type Repository struct {
	// contains filtered or unexported fields
}

Repository is the primary entry point for all backup operations.

func Open

func Open(ctx context.Context, b Backend, pw Passphrase, opts ...Option) (*Repository, error)

Open opens an existing repository in b.

It reads the config object first and returns ErrNotInitialized when the backend holds none, so Open never creates a repository; use Init for that. A config or key file naming a format this build does not read returns ErrUnsupportedFormat.

Then it unseals the master key with pw before any other object is read. It returns ErrWrongPassword when no key file opens, ErrPasswordRequired for WithoutPassword on a protected repository and ErrNotPasswordProtected for a password on a repository created with WithoutPassword. Unsealing a password protected key file runs argon2id with 64 MiB of memory.

func (*Repository) Backup

func (r *Repository) Backup(ctx context.Context, src Source, opts ...BackupOption) (*Snapshot, error)

Backup walks src and stores its entries as a new snapshot.

A cancelled backup leaves orphan chunks and tree objects behind, no snapshot: the manifest is written last, so the snapshot never becomes visible. The orphans stay until Prune sweeps them, and a later backup of the same data reuses them.

func (*Repository) ChangePassword

func (r *Repository) ChangePassword(ctx context.Context, pw Passphrase) error

ChangePassword replaces the key file the repository was opened with by one sealed under pw, which may also be WithoutPassword. The master key stays, so no data is rewritten; for the same reason the change does not revoke access for anyone who could read the master key before, through the old password or a copy of the old key file.

It takes the exclusive lock. The new key file is written before the old one is deleted: when the write fails the old password still opens the repository, when the delete fails both do and the error says so.

func (*Repository) Check

func (r *Repository) Check(ctx context.Context) (*CheckResult, error)

Check verifies every snapshot in the repository.

func (*Repository) Close

func (r *Repository) Close() error

Close releases any resources held by the repository (and the backend).

func (*Repository) Export

func (r *Repository) Export(ctx context.Context, id string, w io.Writer) error

Export writes a portable tar.gz archive of one snapshot to w. id is a full snapshot id or a unique prefix.

func (*Repository) Forget

func (r *Repository) Forget(
	ctx context.Context,
	policy ForgetPolicy,
	filter SnapshotFilter,
	opts ...ForgetOption,
) (*ForgetResult, error)

Forget removes the snapshots that policy does not keep among those matching filter. Chunks stay until Prune.

A policy without a positive rule removes every match, so it is only accepted together with filter.IDs; otherwise Forget returns ErrEmptyPolicy and deletes nothing.

Removing named ids without a policy reads only those manifests, and one that fails with ErrCorruptManifest is still removed, reported with its ID alone. This is how a corrupt manifest leaves the repository.

func (*Repository) ID

func (r *Repository) ID() string

ID returns the repository id written by Init.

func (*Repository) Import

func (r *Repository) Import(ctx context.Context, reader io.Reader) (*Snapshot, error)

Import reads a portable tar.gz archive from reader and re-indexes it.

func (*Repository) Locks

func (r *Repository) Locks(ctx context.Context) ([]LockInfo, error)

Locks lists the lock objects in the repository, newest first, stale ones included.

func (*Repository) Prune

func (r *Repository) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)

Prune removes CAS objects that no snapshot references. Pass WithPruneDryRun to report what would be removed without making changes.

func (*Repository) Restore

func (r *Repository) Restore(ctx context.Context, id string, sink Sink, opts ...RestoreOption) (*RestoreResult, error)

Restore streams the snapshot into sink. id is a full snapshot id or a unique prefix.

func (*Repository) Snapshot

func (r *Repository) Snapshot(ctx context.Context, id string) (*Snapshot, error)

Snapshot returns one snapshot. id is a full snapshot id or a unique prefix.

func (*Repository) Snapshots

func (r *Repository) Snapshots(ctx context.Context, filter SnapshotFilter) ([]*Snapshot, error)

Snapshots returns all snapshots matching filter, newest first.

func (*Repository) Unlock

func (r *Repository) Unlock(ctx context.Context, opts ...UnlockOption) (int, error)

Unlock removes stale locks and returns how many it removed. Pass WithForce to remove every lock, including those that do not open.

func (*Repository) Verify

func (r *Repository) Verify(ctx context.Context, id string) (*VerifyResult, error)

Verify checks one snapshot: manifest, tree and every chunk. id is a full snapshot id or a unique prefix.

func (*Repository) Walk

func (r *Repository) Walk(ctx context.Context, id string, fn func(FileInfo) error) error

Walk calls fn for every entry of a snapshot in tree order, parents before children, without materialising the list. id is a full snapshot id or a unique prefix. An error from fn ends the walk and is returned.

Directory paths carry a trailing slash, the way tar names them; every other entry carries the plain path a Source yielded.

type RestoreOption

type RestoreOption func(*restoreOptions)

RestoreOption is a functional option for Restore.

func WithRestoreCache

func WithRestoreCache(bytes int64) RestoreOption

WithRestoreCache caps the bytes of chunk content the restore keeps around to serve content that repeats within the snapshot. The default is 256 MiB, and zero disables the cache.

The memory a restore holds is this cap plus one chunk per worker in flight, where the workers are bounded by WithConcurrency. It does not grow with the size of the snapshot or of any single file.

func WithRestorePaths

func WithRestorePaths(paths ...string) RestoreOption

WithRestorePaths restores only the entries under the given paths. Prefixes are matched against the snapshot's own paths, and a path that leads nowhere in the snapshot restores nothing. The walk is pruned before any chunk is read, so the skipped part costs nothing.

func WithRestoreProgress

func WithRestoreProgress(fn ProgressFunc) RestoreOption

WithRestoreProgress sets the observer for this restore. One observer per call, so two operations on the same Repository never interleave their events. fn must not block.

type RestoreResult

type RestoreResult struct {
	FilesRestored int
	BytesRestored int64
}

RestoreResult is returned by Restore.

type Sink

type Sink interface {
	Open(ctx context.Context, meta SourceMeta) error
	Receive(ctx context.Context, entry FileInfo, r io.Reader) error
	Close() error
}

Sink receives file entries during a restore.

Lifecycle:

  1. Open — called once before any Receive; Sink may inspect meta and reject incompatible source types by returning an error (wrap ErrIncompatibleSourceType where appropriate).
  2. Receive — called once per entry, in walk order.
  3. Close — called once after the last Receive (or on abort).

The core reconstructs each file's byte stream from the CAS and calls Receive once per entry, parents before children. A FileTypeHardlink entry always arrives after the entry it points at.

Implementations must:

  • validate source type in Open and return ErrIncompatibleSourceType on mismatch
  • handle r = nil for everything but regular files
  • fully consume r for regular files before returning
  • call Close exactly once after the last Receive

type Snapshot

type Snapshot struct {
	ID         string
	SourceName string
	SourceType SourceType
	SourceMeta map[string]string // labels the source recorded, see SourceMeta.Labels
	Host       string
	Tags       []string
	Comment    string
	CreatedAt  time.Time
	ParentID   string
	Stats      SnapshotStats
}

Snapshot describes a completed backup, as returned by the public API.

type SnapshotFilter

type SnapshotFilter struct {
	// IDs selects exactly these snapshots. Snapshots resolves prefixes to
	// full ids before matching, so Match itself compares them verbatim.
	IDs []string

	// Name filters by source name (volume name, path, etc.); empty = all.
	Name string

	// Type filters by source type, e.g. SourceTypeFile; empty = all.
	Type SourceType

	// Tags filters to snapshots carrying ALL the given tags; empty = all.
	Tags []string

	// Host filters by host; empty = all.
	Host string

	// Before includes only snapshots created before this time (exclusive).
	Before time.Time

	// After includes only snapshots created after this time (inclusive).
	After time.Time
}

SnapshotFilter narrows which snapshots are returned by Snapshots or Forget. Zero value means no filtering — all snapshots are included.

func (SnapshotFilter) Match

func (f SnapshotFilter) Match(s *Snapshot) bool

Match returns true if the snapshot matches all non-zero fields of f. IDs are compared verbatim; Snapshots resolves prefixes before calling this.

type SnapshotStats

type SnapshotStats struct {
	FileCount          int           `json:"total_files"`
	DirCount           int           `json:"total_dirs"`
	Skipped            int           `json:"skipped"`
	FilesUnchanged     int           `json:"files_unchanged,omitempty"`
	TotalSize          int64         `json:"total_size"`
	StoredSize         int64         `json:"stored_size"`
	NewChunksRawSize   int64         `json:"new_chunks_raw_size"`
	ChunksTotal        int           `json:"chunks_total"`
	ChunksNew          int           `json:"chunks_new"`
	ChunksReused       int           `json:"chunks_reused"`
	DeduplicationRatio float64       `json:"deduplication_ratio"`
	CompressionRatio   float64       `json:"compression_ratio"`
	Duration           time.Duration `json:"-"`
}

SnapshotStats holds deduplication and compression statistics for a snapshot. Skipped counts entries the source yielded that the v1 format does not model: device nodes, fifos and sockets. FilesUnchanged counts the regular files a backup took from the parent snapshot without reading them.

func (SnapshotStats) MarshalJSON

func (s SnapshotStats) MarshalJSON() ([]byte, error)

MarshalJSON writes the manifest shape: Duration becomes the string field "duration".

func (*SnapshotStats) UnmarshalJSON

func (s *SnapshotStats) UnmarshalJSON(data []byte) error

UnmarshalJSON reads the manifest shape. An unparsable duration leaves the field zero rather than failing the whole manifest.

type Source

type Source interface {
	Meta() SourceMeta
	Walk(ctx context.Context, fn WalkFunc) error
}

Source drives the backup pipeline entry by entry.

Entries may arrive in any order; the core sorts every tree node by name before hashing it.

Implementations must:

  • yield a hardlink target before the FileTypeHardlink entries naming it
  • pass open = nil for everything but regular files
  • for regular files, pass an open that yields at most entry.Size bytes and does no work until it is called; a walk that never calls open must succeed
  • be safe for a single concurrent Walk call

type SourceMeta

type SourceMeta struct {
	// Name is the logical name used as the snapshot label (e.g. volume name, path).
	Name string

	// Type identifies the origin of the data.
	Type SourceType

	// Labels holds arbitrary key-value metadata (e.g. docker_host, docker_version).
	Labels map[string]string
}

SourceMeta is stored verbatim in the snapshot manifest.

type SourceType

type SourceType string

SourceType tags a snapshot with its origin. Core defines only SourceTypeFile. Plugin packages (e.g. pkg/arkdocker) define their own SourceType constants; the string value is stored verbatim in the snapshot manifest and compared at restore time.

const SourceTypeFile SourceType = "file"

SourceTypeFile identifies a local filesystem backup.

type UnlockOption

type UnlockOption func(*unlockConfig)

UnlockOption configures an Unlock call.

func WithForce

func WithForce() UnlockOption

WithForce makes Unlock remove every lock, not only the stale ones. It breaks locks that running operations still hold, so the caller has to know that no such operation exists.

type VerifyResult

type VerifyResult struct {
	SnapshotID    string
	ChunksChecked int
	MissingChunks []string
	CorruptChunks []string
	OK            bool
}

VerifyResult is returned by Verify.

type WalkFunc

type WalkFunc func(entry FileInfo, open func() (io.ReadCloser, error)) error

WalkFunc is called for each entry produced by a Source.

open is nil for every entry except regular files; the caller must check entry.Type. For a regular file, open returns a reader for entry.Size bytes. The caller calls open at most once and closes the reader before returning, or never calls it when the parent snapshot already holds the file unchanged. A ConcurrentSource lets the caller open and read the file after returning instead. The next call to fn advances the source past the entry, read or not (archive/tar.Reader.Next skips unread bytes).

Directories

Path Synopsis
arkdocker module
Package arkfs implements ark.Source and ark.Sink for local filesystem directories.
Package arkfs implements ark.Source and ark.Sink for local filesystem directories.
Package arklocal implements ark.Backend on the local filesystem.
Package arklocal implements ark.Backend on the local filesystem.
Package arkmem implements ark.Backend in memory.
Package arkmem implements ark.Backend in memory.
Package arksftp implements ark.Backend over an SFTP connection.
Package arksftp implements ark.Backend over an SFTP connection.
Package arktest holds the conformance suites every ark implementation runs.
Package arktest holds the conformance suites every ark implementation runs.
Package arkuri opens an ark.Backend from a single configuration string.
Package arkuri opens an ark.Backend from a single configuration string.
Package chunker implements content-defined chunking (CDC) using the FastCDC algorithm with gear hashing.
Package chunker implements content-defined chunking (CDC) using the FastCDC algorithm with gear hashing.
scripts
bench command
Command bench builds the seeded benchmark dataset that ark is timed against restic on: gen writes it, mutate applies the incremental change set in place, hash prints a digest of a tree, run times ark against restic.
Command bench builds the seeded benchmark dataset that ark is timed against restic on: gen writes it, mutate applies the incremental change set in place, hash prints a digest of a tree, run times ark against restic.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL