cloudstic

package module
v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 6, 2026 License: MIT Imports: 12 Imported by: 0

README

Cloudstic CLI

Content-addressable, encrypted backup tool for Google Drive, OneDrive, and local files.

Features

  • Encrypted by default: AES-256-GCM encryption with password, platform key, or recovery key slots
  • Content-addressable storage: Deduplication across sources; identical files stored only once
  • Incremental backups: Only changed files are stored
  • Multiple sources: Google Drive, Google Drive Changes API, OneDrive, local directories
  • Multiple backends: Local filesystem, Amazon S3 (and compatibles like R2, MinIO), or Backblaze B2
  • Retention policies: Keep-last, hourly, daily, weekly, monthly, yearly
  • Point-in-time restore: Restore any snapshot, any file, any time

Supported Sources

Source Flag Description
Local directory -source local Back up any local folder
Google Drive -source gdrive Full rescan of My Drive or a Shared Drive
Google Drive (Changes) -source gdrive-changes Recommended. Fast incremental backup via the Changes API
OneDrive -source onedrive Full scan of a Microsoft OneDrive account
OneDrive (Changes) -source onedrive-changes Recommended. Fast incremental backup via the Delta API

Google Drive and OneDrive work out of the box. On first run, Cloudstic opens your browser for authorization and caches the token locally. See the User Guide — Sources for details.

Install

brew install cloudstic/tap/cloudstic   # macOS / Linux
winget install Cloudstic.CLI           # Windows
go install github.com/cloudstic/cli/cmd/cloudstic@latest  # with Go

Or download a binary from Releases. See the User Guide for all options.

Quick Start

# Initialize an encrypted repository (prompts for password interactively)
cloudstic init

# Back up a local directory (prompts for password if not set via flag or env)
cloudstic backup -source local -source-path ~/Documents

# Back up Google Drive (opens browser for auth on first run)
cloudstic backup -source gdrive-changes

# List snapshots
cloudstic list

# Restore latest snapshot to a zip file
cloudstic restore

# Preview what a backup would do (dry run)
cloudstic backup -source local -source-path ~/Documents -dry-run

When running interactively, Cloudstic prompts for the repository password if no credential is provided via flags or environment variables. For non-interactive use (scripts, cron), pass -encryption-password or set CLOUDSTIC_ENCRYPTION_PASSWORD:

cloudstic init -encryption-password "my passphrase"
cloudstic backup -source local -source-path ~/Documents -encryption-password "my passphrase"

Performance

Benchmarked against Restic, Borg, and Duplicacy on a ~1 GB dataset (local) and a real Google Drive account (~40 MB). Full methodology and numbers in docs/benchmark-results.md.

Local filesystem (time / peak RAM):

Operation Cloudstic Restic Borg Duplicacy
Initial backup 0.61s / 259 MB 1.80s / 314 MB 1.69s / 139 MB 3.44s / 284 MB
Incremental (no changes) 0.05s / 96 MB 0.77s / 72 MB 0.67s / 72 MB 0.04s / 44 MB
Add 200 MB new data 0.14s / 152 MB 1.07s / 283 MB 0.57s / 136 MB 0.93s / 195 MB

Google Drive — each step run stateless (no rclone cache) to reflect real-world cold-start conditions:

Operation Cloudstic Restic Borg
Initial backup 6.08s 11.14s 15.06s
Incremental (no changes) 0.56s 14.70s 25.49s

Cloudstic uses the Google Drive Changes API natively — incremental backups only fetch what actually changed, no full re-scan required. Restic and Borg rely on rclone FUSE mounts and must re-download the entire dataset to detect changes.

Documentation

Full documentation is available at docs.cloudstic.com.

This repository also contains developer-focused reference docs:

  • User Guide: commands, setup, encryption, retention policies
  • Source API: source interface, implementations, and how to add a new source
  • Specification: object types, backup/restore flow, HAMT structure
  • Encryption: key slot design, AES-256-GCM, recovery keys
  • Storage Model: content-addressable storage layout
  • Contributing: testing, profiling, debugging

Cloud Service

Don't want to manage infrastructure? Cloudstic Cloud handles scheduling, storage, and retention automatically. Same engine, zero ops.

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	WithInitCredentials  = engine.WithInitCredentials
	WithInitRecovery     = engine.WithInitRecovery
	WithInitNoEncryption = engine.WithInitNoEncryption
	WithInitAdoptSlots   = engine.WithInitAdoptSlots
)
View Source 
var (
	WithVerbose      = engine.WithVerbose
	WithBackupDryRun = engine.WithBackupDryRun
	WithTags         = engine.WithTags
	WithGenerator    = engine.WithGenerator
	WithMeta         = engine.WithMeta
	WithExcludeHash  = engine.WithExcludeHash
)
View Source 
var (
	WithRestoreDryRun  = engine.WithRestoreDryRun
	WithRestoreVerbose = engine.WithRestoreVerbose
	WithRestorePath    = engine.WithRestorePath
)
View Source 
var (
	WithPruneDryRun  = engine.WithPruneDryRun
	WithPruneVerbose = engine.WithPruneVerbose
)
View Source 
var (
	WithPrune         = engine.WithPrune
	WithDryRun        = engine.WithDryRun
	WithForgetVerbose = engine.WithForgetVerbose
	WithKeepLast      = engine.WithKeepLast
	WithKeepHourly    = engine.WithKeepHourly
	WithKeepDaily     = engine.WithKeepDaily
	WithKeepWeekly    = engine.WithKeepWeekly
	WithKeepMonthly   = engine.WithKeepMonthly
	WithKeepYearly    = engine.WithKeepYearly
	WithGroupBy       = engine.WithGroupBy
	WithFilterTag     = engine.WithFilterTag
	WithFilterSource  = engine.WithFilterSource
	WithFilterAccount = engine.WithFilterAccount
	WithFilterPath    = engine.WithFilterPath
)
View Source 
var (
	WithReadData     = engine.WithReadData
	WithCheckVerbose = engine.WithCheckVerbose
	WithSnapshotRef  = engine.WithSnapshotRef
)
View Source 
var WithDiffVerbose = engine.WithDiffVerbose
View Source 
var WithListVerbose = engine.WithListVerbose
View Source 
var WithLsVerbose = engine.WithLsVerbose

Functions

func AddRecoveryKey added in v1.7.0 

func AddRecoveryKey(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain) (string, error)

AddRecoveryKey generates a BIP39 recovery key for the repository, authenticating with kc to obtain the master key. Returns the 24-word mnemonic phrase.

func ChangePassword added in v1.7.0 

func ChangePassword(ctx context.Context, rawStore store.ObjectStore, kc keychain.Chain, pwd PasswordProvider) error

ChangePassword replaces the password key slot using the provided keychain to authenticate and newPassword as the new passphrase.

Types

type BackupOption 

type BackupOption = engine.BackupOption

type BackupResult added in v1.2.0 

type BackupResult = engine.RunResult

type CatResult added in v1.4.6 

type CatResult struct {
	Key  string // The object key requested
	Data []byte // Raw object data (typically JSON)
}

CatResult contains the raw data for an object key.

type CheckError added in v1.4.7 

type CheckError = engine.CheckError

type CheckOption added in v1.4.7 

type CheckOption = engine.CheckOption

type CheckResult added in v1.4.7 

type CheckResult = engine.CheckResult

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the high-level interface for using Cloudstic as a library.

func NewClient 

func NewClient(base store.ObjectStore, opts ...ClientOption) (*Client, error)

func (*Client) Backup 

func (c *Client) Backup(ctx context.Context, src source.Source, opts ...BackupOption) (*BackupResult, error)

func (*Client) BreakLock added in v1.3.0 

func (c *Client) BreakLock(ctx context.Context) ([]*RepoLock, error)

func (*Client) Cat added in v1.4.6 

func (c *Client) Cat(ctx context.Context, keys ...string) ([]*CatResult, error)

Cat fetches the raw data for one or more object keys from the repository. Object keys can be snapshot/<hash>, filemeta/<hash>, content/<hash>, node/<hash>, chunk/<hash>, config, index/latest, keys/<slot>, etc.

This is useful for debugging, inspection, and understanding the internal structure of the repository.

func (*Client) Check added in v1.4.7 

func (c *Client) Check(ctx context.Context, opts ...CheckOption) (*CheckResult, error)

Check verifies the integrity of the repository by walking the full reference chain (snapshots → HAMT nodes → filemeta → content → chunks) and checking that every referenced object can be read. With WithReadData(), chunk data is re-hashed for byte-level verification.

func (*Client) Diff 

func (c *Client) Diff(ctx context.Context, snap1, snap2 string, opts ...DiffOption) (*DiffResult, error)

func (*Client) Forget 

func (c *Client) Forget(ctx context.Context, snapshotID string, opts ...ForgetOption) (*ForgetResult, error)

func (*Client) ForgetPolicy 

func (c *Client) ForgetPolicy(ctx context.Context, opts ...ForgetOption) (*PolicyResult, error)

func (*Client) List 

func (c *Client) List(ctx context.Context, opts ...ListOption) (*ListResult, error)

func (*Client) LsSnapshot 

func (c *Client) LsSnapshot(ctx context.Context, snapshotID string, opts ...LsSnapshotOption) (*LsSnapshotResult, error)

func (*Client) Prune 

func (c *Client) Prune(ctx context.Context, opts ...PruneOption) (*PruneResult, error)

func (*Client) Restore 

func (c *Client) Restore(ctx context.Context, w io.Writer, snapshotRef string, opts ...RestoreOption) (*RestoreResult, error)

Restore writes the snapshot's file tree as a ZIP archive to w. snapshotRef can be "", "latest", a bare hash, or "snapshot/<hash>".

func (*Client) Store added in v1.1.0 

func (c *Client) Store() store.ObjectStore

type ClientOption 

type ClientOption func(*Client)

ClientOption configures a Client.

func WithEncryptionKey added in v1.1.0 

func WithEncryptionKey(key []byte) ClientOption

WithEncryptionKey directly sets the AES-256-GCM encryption key (32 bytes). This bypasses repo config detection and unconditionally applies encryption. The HMAC deduplication key is automatically derived from this key. Use this for the SaaS product where the key is already resolved externally.

func WithKeychain added in v1.9.0 

func WithKeychain(kc keychain.Chain) ClientOption

WithKeychain sets a Keychain for automatic master key resolution. During NewClient, the repo config is read from the store; if the repository is encrypted, Resolve is called to obtain the master key and the encryption key is derived. If the repository is not encrypted, the keychain is silently ignored.

func WithPackfile added in v1.4.3 

func WithPackfile(enable bool) ClientOption

WithPackfile enables bundling small objects into 8MB packs to save API calls.

func WithReporter 

func WithReporter(r Reporter) ClientOption

WithReporter sets the progress reporter for the client.

type DiffOption 

type DiffOption = engine.DiffOption

type DiffResult 

type DiffResult = engine.DiffResult

type ForgetOption 

type ForgetOption = engine.ForgetOption

type ForgetResult added in v1.2.0 

type ForgetResult = engine.ForgetResult

type InitOption added in v1.7.0 

type InitOption = engine.InitOption

type InitResult added in v1.7.0 

type InitResult = engine.InitResult

func InitRepo added in v1.7.0 

func InitRepo(ctx context.Context, rawStore store.ObjectStore, opts ...InitOption) (*InitResult, error)

InitRepo bootstraps a new repository on the given raw (undecorated) store. This is a package-level function because init runs before the full Client decorator chain (encryption, compression, packfiles) is set up.

type KMSClient added in v1.9.0 

type KMSClient = crypto.KMSClient

KMSClient is re-exported for callers that provide KMS credentials.

type KeySlot added in v1.4.6 

type KeySlot = keychain.KeySlot

KeySlot is re-exported for callers that need to inspect slot metadata.

func ListKeySlots added in v1.7.0 

func ListKeySlots(ctx context.Context, rawStore store.ObjectStore) ([]KeySlot, error)

ListKeySlots returns all encryption key slots in the repository. Returns an error if the repository is not initialized or not encrypted.

type ListOption 

type ListOption = engine.ListOption

type ListResult 

type ListResult = engine.ListResult

type LsSnapshotOption 

type LsSnapshotOption = engine.LsSnapshotOption

type LsSnapshotResult 

type LsSnapshotResult = engine.LsSnapshotResult

type PasswordProvider added in v1.7.0 

type PasswordProvider interface {
	NewPassword(ctx context.Context) (string, error)
}

PasswordProvider supplies a new password when prompted. It is used by ChangePassword to obtain the replacement passphrase. Implementations may prompt the user interactively, derive a password programmatically, or return a static value.

type PasswordProviderFunc added in v1.7.0 

type PasswordProviderFunc func(ctx context.Context) (string, error)

PasswordProviderFunc is a function adapter for PasswordProvider. Any func(context.Context) (string, error) can be used as a PasswordProvider: 

client.ChangePassword(ctx, store, creds, cloudstic.PasswordProviderFunc(func(ctx context.Context) (string, error) {
	return promptUser("New password: ")
}))

func (PasswordProviderFunc) NewPassword added in v1.7.0 

func (f PasswordProviderFunc) NewPassword(ctx context.Context) (string, error)

type PasswordString added in v1.7.0 

type PasswordString string

PasswordString is a PasswordProvider that returns a fixed string. Use this when the new password is already known at call time: 

client.ChangePassword(ctx, store, creds, cloudstic.PasswordString("my-new-password"))

func (PasswordString) NewPassword added in v1.7.0 

func (p PasswordString) NewPassword(ctx context.Context) (string, error)

type Phase added in v1.2.0 

type Phase = ui.Phase

Phase represents an active progress tracking phase.

type PolicyResult 

type PolicyResult = engine.PolicyResult

type PruneOption 

type PruneOption = engine.PruneOption

type PruneResult added in v1.2.0 

type PruneResult = engine.PruneResult

type RepoConfig added in v1.2.0 

type RepoConfig = core.RepoConfig

RepoConfig is the repository marker written by "init".

func LoadRepoConfig added in v1.7.0 

func LoadRepoConfig(ctx context.Context, rawStore store.ObjectStore) (*RepoConfig, error)

LoadRepoConfig reads the repository marker from a raw (undecorated) store. Returns (nil, nil) if the repository has not been initialized yet. Returns an error if the store is unreachable (e.g. invalid credentials).

type RepoLock added in v1.3.0 

type RepoLock = engine.RepoLock

type Reporter added in v1.2.0 

type Reporter = ui.Reporter

Reporter defines the interface for progress reporting.

type RestoreOption 

type RestoreOption = engine.RestoreOption

type RestoreResult 

type RestoreResult = engine.RestoreResult

Source Files

View all Source files

Directories

Path Synopsis
cmd
cloudstic command
internal
core
engine
hamt
logger
paths
retry
sftp
ui
pkg
crypto
Package crypto provides authenticated encryption primitives for backup data.
 Package crypto provides authenticated encryption primitives for backup data.
keychain
source
store

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.