goback

README

goback

Deduplicating, encrypted backups for one user and one store. An agent backs directories up into a store: a local directory, a bucket on Google Cloud Storage or anything S3-compatible, or a goback server that gathers several agents into one store.

How it stores data

• Objects are content-addressed by SHA-256, like git's: a commit points at a tree, a tree at files and subtrees, a file at the chunks it is made of.
• Files are cut into chunks with FastCDC, so an edit stores only the chunks it changed, and the same chunk is stored once across files, commits and sets.
• Objects are packed into archives, each with an index. A local SQLite or Postgres index answers queries, and can always be rebuilt from the archives.
• A backup only uploads what the store doesn't already hold, and a restore only downloads what the target directory doesn't already hold.

Encryption

• Store key (client side). With --store-key, names and contents are encrypted before they leave the agent; a server never sees them. The key can be escrowed in the store under a passphrase (goback key escrow) and opened with --passphrase. Without the key or the passphrase, nothing can be restored.
• At-rest key (server side). With --at-rest-key, archives are sealed where they are stored, independently of the store key.

Quick start

go build ./cmd/goback

goback key new my-store                 # writes store.key; keep a copy elsewhere

export GOBACK_STORAGE=/backups/store GOBACK_INDEX=/backups/index

goback --store-key store.key --set home commit new ~/

goback --store-key store.key --set home commit list

goback --store-key store.key --set home \
  commit restore ~/restored             # the latest commit; add an age like 24h for an older one

--storage also takes gcs://bucket, s3://bucket?endpoint=…, or goback://<secret>@host:port for a goback server. --storage and --index (or GOBACK_STORAGE and GOBACK_INDEX) have no defaults; a command that needs one says so. --cache-dir (or GOBACK_CACHE_DIR) names a per-machine directory for every cache: a backup skips unchanged files with it and a restore downloads less.

Commands

Command	Does
agent	run a machine's backups on their schedules from one config file
commit	back a directory up, list commits, restore, delete and undelete
file	list and restore single files and their versions
set	delete and undelete a set, show or change its retention
pin	keep a commit regardless of retention
key	make, escrow and recover store keys; make at-rest keys
gc	mark what live commits and pins reach, and rewrite archives that are mostly dead
maintain	run the housekeeping that is due: retire, compact, build presence filters, and collect garbage weekly
scrub, repair	rehash every stored object, and rewrite the archives holding corrupt ones
fix	rebuild the index from the store
index	index commits the store holds without a full rebuild
object	read stored objects by ref
postgres	back up a Postgres cluster with point-in-time recovery
mysql	back up a MySQL server with optional point-in-time recovery
server	serve one store to many agents over gRPC, with TLS and an operator API

goback <command> --help describes each one.

Retention

A set keeps the last commit of each calendar hour, day, week or month, by brackets counted back from now, newest first:

goback set retention home \
  --keep hourly=336 --keep daily=60 --keep weekly=12 --keep monthly

Commits the policy lets go are tombstoned by the next retirement; a commit deleted by hand waits in the trash first, where it can be taken back. Once no live commit or pin reaches an object, garbage collection drops it.

Without a server, run goback maintain from cron, say hourly: it retires and compacts every time and collects garbage once the last collection is a week old.

A server for several agents

goback --storage gcs://my-bucket --index /var/lib/goback/index \
  server --address :6060 --secret-file secret --tls-cert cert.pem --tls-key key.pem

The server listens on 127.0.0.1:6060 unless --address says otherwise. Every agent presents the shared secret and its own --agent-id. The server runs maintenance itself: it sweeps, compacts, retires and collects on its own intervals.

The agent

goback agent run --config goback.yaml runs a machine's backups on their schedules: directories, Postgres clusters and MySQL servers, listed once with the store they go to. goback agent check validates the file, opens the store and lists what each database lacks; goback agent status --check is a health check. contrib/systemd runs it as a service.

Postgres

goback postgres takes base backups and streams the WAL through a replication slot into two sets, and restores a cluster to any moment in between. contrib/postgres runs it with Docker Compose.

MySQL

goback mysql streams mysqldump into one set and, optionally, receives the binary log as a replica into another, and restores a server to a dump or to any second after it. contrib/mysql runs it with Docker Compose.

As a Go library

Package github.com/gobackio/goback opens a store from the same settings the CLI takes; the CLI is built on it. The API is not stable yet and may change between minor versions.

repo, err := goback.Open(ctx, goback.Settings{
	Storage:  "/backups/store",
	Index:    "/backups/index",
	StoreKey: "store.key",
})
if err != nil {
	return err
}
defer repo.Close()

env, err := repo.Env(ctx, goback.Caches{})
if err != nil {
	return err
}
defer env.Close()

result, err := job.Path(ctx, env, job.PathSettings{Set: "home", Dir: "/home/me"})

backup/job runs each kind of backup once, agent runs them on schedules, backup.Restorer restores, storage/maintenance runs the housekeeping, and Repo.Serve serves a store as goback server does.

Development

go build ./...
go test ./...

Some tests start Postgres, MySQL and MinIO in Docker through testcontainers.

License

Apache License 2.0; see LICENSE.

Documentation

Overview

Package goback opens a store from the location the goback CLI takes: its objects, its index, its key and the caches under a cache directory. The API is not stable yet and may change between minor versions.

Index

• Constants
• type Caches
• type Repo
  • func Open(ctx context.Context, s Settings) (*Repo, error)
  • func (r *Repo) BlobCache(ctx context.Context) (*blobcache.Cache, error)
  • func (r *Repo) Close() error
  • func (r *Repo) Compact(ctx context.Context) (*pack.CompactReport, error)
  • func (r *Repo) Env(ctx context.Context, c Caches) (env *job.Env, err error)
  • func (r *Repo) Key(ctx context.Context) (*storekey.Key, error)
  • func (r *Repo) LatestCommit(ctx context.Context, set string) (*proto.Ref, error)
  • func (r *Repo) Serve(ctx context.Context, listener net.Listener, opts ServeOptions) error
  • func (r *Repo) StoreCache(name string) string
• type ServeOptions
• type Settings

Examples

• Open

Constants

const (
	DefaultMetadataCacheSize = 1 << 30
	DefaultBlobCacheSize     = 4_000_000_000
)

The cache sizes a zero Settings field stands for.

Types

type Caches

type Caches struct {
	// Stats and Objects let a path backup skip unchanged files and trees.
	Stats, Objects bool
	// Trees keeps the commits and trees a log capture reads every commit.
	Trees bool
	// Blobs keeps every blob a path backup uploads, for restores to read.
	Blobs bool
}

Caches picks the caches Env opens under Settings.CacheDir; without a cache directory it opens none.

type Repo

type Repo struct {
	Store backup.ObjectStore
	Index backup.Index
	// contains filtered or unexported fields
}

Repo is an open store. Close it when done.

func Open

func Open(ctx context.Context, s Settings) (*Repo, error)

Open opens the store s names and its index.

Example

A program backs a directory up into a store in a local directory and restores the commit it made into another.

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"path/filepath"
	"time"

	"github.com/gobackio/goback"
	"github.com/gobackio/goback/backup"
	"github.com/gobackio/goback/backup/job"
)

func main() {
	dir, err := os.MkdirTemp("", "goback-example")
	if err != nil {
		log.Fatal(err)
	}
	defer os.RemoveAll(dir)

	home := filepath.Join(dir, "home")
	if err := os.MkdirAll(home, 0o700); err != nil {
		log.Fatal(err)
	}

	if err := os.WriteFile(filepath.Join(home, "notes.md"), []byte("hello\n"), 0o600); err != nil {
		log.Fatal(err)
	}

	ctx := context.Background()

	repo, err := goback.Open(ctx, goback.Settings{
		Storage: filepath.Join(dir, "store"),
		Index:   filepath.Join(dir, "index"),
	})
	if err != nil {
		log.Fatal(err)
	}
	defer repo.Close()

	env, err := repo.Env(ctx, goback.Caches{})
	if err != nil {
		log.Fatal(err)
	}
	defer env.Close()

	if _, err := job.Path(ctx, env, job.PathSettings{Set: "home", Dir: home, ProgressInterval: -1}); err != nil {
		log.Fatal(err)
	}

	commit, err := backup.PickCommit(ctx, repo.Index, "home", nil, time.Now())
	if err != nil {
		log.Fatal(err)
	}

	restorer := &backup.Restorer{Store: repo.Store, Key: env.Key}

	restored := filepath.Join(dir, "restored")
	if err := restorer.RestoreTree(ctx, commit, restored, backup.TreeOptions{}); err != nil {
		log.Fatal(err)
	}

	data, err := os.ReadFile(filepath.Join(restored, "notes.md"))
	if err != nil {
		log.Fatal(err)
	}

	fmt.Print(string(data))
}

Output:
hello

func (*Repo) BlobCache

func (r *Repo) BlobCache(ctx context.Context) (*blobcache.Cache, error)

BlobCache opens the cache of the blobs backups upload and restores read, under Settings.CacheDir and apart per store key, or returns nil when nothing is cached.

func (*Repo) Close

func (r *Repo) Close() error

Close closes the store, then its index, which a directory or bucket store's last writes still go to.

func (*Repo) Compact

func (r *Repo) Compact(ctx context.Context) (*pack.CompactReport, error)

Compact rewrites a directory or bucket store's sparse archives; it does nothing for a store server, which compacts its own.

func (*Repo) Env

func (r *Repo) Env(ctx context.Context, c Caches) (env *job.Env, err error)

Env is the job environment of the repo: its store, index and key, the agent id, and the caches c names. Close it before the repo.

func (*Repo) Key

func (r *Repo) Key(ctx context.Context) (*storekey.Key, error)

Key is the store key: loaded from Settings.StoreKey, else recovered with Settings.Passphrase from the copy the store keeps escrowed, else nil for a store written in the clear. It is read once, on the first call.

func (*Repo) LatestCommit

func (r *Repo) LatestCommit(ctx context.Context, set string) (*proto.Ref, error)

LatestCommit is the newest commit of set, a set name or a numeric set id: an id is read from the set's head, a name from the index, falling back to the heads the store keeps when the index has none.

func (*Repo) Serve

func (r *Repo) Serve(ctx context.Context, listener net.Listener, opts ServeOptions) error

Serve serves the repo to agents over gRPC on listener, as goback server does: the store API, the Admin service when the index keeps sets and policy, health checks, and the maintenance jobs of opts.Schedule. It returns once ctx ends, after the calls in flight finish, or when the listener fails.

func (*Repo) StoreCache

func (r *Repo) StoreCache(name string) string

StoreCache is the directory under Settings.CacheDir that holds this store's caches of that name, or "" when nothing is cached.

type ServeOptions

type ServeOptions struct {
	// Secret is what every agent presents; Serve refuses an empty one.
	Secret string
	// TLS is what the server presents to agents. Without it Serve listens
	// in plaintext only when PlaintextBehindProxy says a proxy in front
	// terminates TLS.
	TLS                  *tls.Config
	PlaintextBehindProxy bool
	// Schedule is how often each maintenance job runs, as is: a zero
	// interval never runs its job. maintenance.DefaultSchedule is what
	// goback server runs by default.
	Schedule maintenance.Schedule
}

ServeOptions say how Serve serves a store.

type Settings

type Settings struct {
	// Storage is the store's location: a directory, gs:// or s3:// with
	// the bucket's query parameters, or goback:// and goback+insecure://
	// for a store server.
	Storage string
	// Index is where a directory or bucket store keeps its index: a
	// directory for SQLite, or a postgres:// URL. A store server brings
	// its own.
	Index string
	// AgentID is who commits and who a store server is told is calling;
	// empty means the hostname.
	AgentID string
	// StoreKey is the store key's file. Without it, Passphrase opens the
	// copy the store keeps escrowed; without either, the store is written
	// in the clear.
	StoreKey   string
	Passphrase string
	// AtRestKey is the file of the key a directory or bucket store
	// encrypts its archives with.
	AtRestKey string
	// CacheDir holds every cache, apart per store; empty caches nothing.
	CacheDir string
	// MetadataCacheSize bounds a bucket's metadata cache in bytes; zero
	// means DefaultMetadataCacheSize and a negative size no bound.
	MetadataCacheSize int64
	// BlobCacheSize is what BlobCache is trimmed to in bytes; zero means
	// DefaultBlobCacheSize and a negative size no bound.
	BlobCacheSize int64
	// Logger is where opening reports; nil means slog.Default.
	Logger *slog.Logger
}

Settings say where a store lives and how to open it. Storage is required, and Index for a directory or bucket store; the rest may be left zero.