litestream

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 21, 2021 License: GPL-3.0 Imports: 26 Imported by: 9

README

Litestream test

Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incrementally to another file or S3. Litestream only communicates with SQLite through the SQLite API so it will not corrupt your database.

If you find this project interesting, please consider starring the project on GitHub.

Installation

Homebrew

TODO

Linux (Debian)

You can download the .deb file from the Releases page page and then run the following:

$ sudo dpkg -i litestream-v0.3.0-linux-amd64.deb

Once installed, you'll need to enable & start the service:

$ sudo systemctl enable litestream
$ sudo systemctl start litestream
Release binaries

You can also download the release binary for your system from the Releases page and run it as a standalone application.

Configuration

Once installed locally, you'll need to create a config file. By default, the config file lives at /etc/litestream.yml but you can pass in a different path to any litestream command using the -config PATH flag.

The configuration specifies one or more dbs and a list of one or more replica locations for each db. Below are some common configurations:

Replicate to S3

This will replicate the database at /path/to/db to the "/db" path inside the S3 bucket named "mybkt".

access-key-id:     AKIAxxxxxxxxxxxxxxxx
secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx

dbs:
  - path: /path/to/db
    replicas:
      - path: s3://mybkt/db
Replicate to another file path

This will replicate the database at /path/to/db to a directory named /path/to/replica.

dbs:
  - path: /path/to/db
    replicas:
      - path: /path/to/replica
Other configuration options

These are some additional configuration options available on replicas:

  • type—Specify the type of replica ("file" or "s3"). Derived from "path".
  • name—Specify an optional name for the replica if you are using multiple replicas.
  • path—File path or URL to the replica location.
  • retention—Length of time to keep replicated WAL files. Defaults to 24h.
  • retention-check-interval—Time between retention enforcement checks. Defaults to 1h.
  • validation-interval—Interval between periodic checks to ensure restored backup matches current database. Disabled by default.

These replica options are only available for S3 replicas:

  • bucket—S3 bucket name. Derived from "path".
  • region—S3 bucket region. Looked up on startup if unspecified.
  • sync-interval—Replication sync frequency.

Usage

Replication

Once your configuration is saved, you'll need to begin replication. If you installed the .deb file then run:

$ sudo systemctl restart litestream

To run litestream on its own, run:

# Replicate using the /etc/litestream.yml configuration.
$ litestream replicate

# Replicate using a different configuration path.
$ litestream replicate -config /path/to/litestream.yml

The litestream command will initialize and then wait indefinitely for changes. You should see your destination replica path is now populated with a generations directory. Inside there should be a 16-character hex generation directory and inside there should be snapshots & WAL files. As you make changes to your source database, changes will be copied over to your replica incrementally.

Restoring a backup

Litestream can restore a previous snapshot and replay all replicated WAL files. By default, it will restore up to the latest WAL file but you can also perform point-in-time restores.

A database can only be restored to a path that does not exist so you don't need to worry about accidentally overwriting your current database.

# Restore database to original path.
$ litestream restore /path/to/db

# Restore database to a new location.
$ litestream restore -o /tmp/mynewdb /path/to/db

# Restore database to a specific point-in-time.
$ litestream restore -timestamp 2020-01-01T00:00:00Z /path/to/db

Point-in-time restores only have the resolution of the timestamp of the WAL file itself. By default, litestream will start a new WAL file every minute so point-in-time restores are only accurate to the minute.

How it works

SQLite provides a WAL (write-ahead log) journaling mode which writes pages to a -wal file before eventually being copied over to the original database file. This copying process is known as checkpointing. The WAL file works as a circular buffer so when the WAL reaches a certain size then it restarts from the beginning.

Litestream works by taking over the checkpointing process and controlling when it is restarted to ensure that it copies every new page. Checkpointing is only allowed when there are no read transactions so Litestream maintains a long-running read transaction against each database until it is ready to checkpoint.

The SQLite WAL file is copied to a separate location called the shadow WAL which ensures that it will not be overwritten by SQLite. This shadow WAL acts as a temporary buffer so that replicas can replicate to their destination (e.g. another file path or to S3). The shadow WAL files are removed once they have been fully replicated. You can find the shadow directory as a hidden directory next to your database file. If you database file is named /var/lib/my.db then the shadow directory will be /var/lib/.my.db-litestream.

Litestream groups a snapshot and all subsequent WAL changes into "generations". A generation is started on initial replication of a database and a new generation will be started if litestream detects that the WAL replication is no longer contiguous. This can occur if the litestream process is stopped and another process is allowed to checkpoint the WAL.

Open-source, not open-contribution

Similar to SQLite, litestream is open source but closed to contributions. This keeps the code base free of proprietary or licensed code but it also helps me continue to maintain and build litestream.

As the author of BoltDB, I found that accepting and maintaining third party patches contributed to my burn out and I eventually archived the project. Writing databases & low-level replication tools involves nuance and simple one line changes can have profound and unexpected changes in correctness and performance. Small contributions typically required hours of my time to properly test and validate them.

I am grateful for community involvement, bug reports, & feature requests. I do not wish to come off as anything but welcoming, however, I've made the decision to keep this project closed to contributions for my own mental health and long term viability of the project.

Documentation

Index

Constants

View Source 
const (
	DefaultMonitorInterval    = 1 * time.Second
	DefaultCheckpointInterval = 1 * time.Minute
	DefaultMinCheckpointPageN = 1000
	DefaultMaxCheckpointPageN = 10000
)

Default DB settings.

View Source 
const (
	WALHeaderChecksumOffset      = 24
	WALFrameHeaderChecksumOffset = 16
)

SQLite WAL constants

View Source 
const (
	MetaDirSuffix = "-litestream"

	WALDirName  = "wal"
	WALExt      = ".wal"
	SnapshotExt = ".snapshot"

	GenerationNameLen = 16
)

Naming constants.

View Source 
const (
	CheckpointModePassive  = "PASSIVE"
	CheckpointModeFull     = "FULL"
	CheckpointModeRestart  = "RESTART"
	CheckpointModeTruncate = "TRUNCATE"
)

SQLite checkpoint modes.

View Source 
const (
	// WALHeaderSize is the size of the WAL header, in bytes.
	WALHeaderSize = 32

	// WALFrameHeaderSize is the size of the WAL frame header, in bytes.
	WALFrameHeaderSize = 24
)
View Source 
const (
	DefaultRetention              = 24 * time.Hour
	DefaultRetentionCheckInterval = 1 * time.Hour
)

Default file replica settings.

View Source 
const BusyTimeout = 1 * time.Second

BusyTimeout is the timeout to wait for EBUSY from SQLite.

Variables

View Source 
var (
	ErrNoSnapshots      = errors.New("no snapshots available")
	ErrChecksumMismatch = errors.New("invalid replica, checksum mismatch")
)

Litestream errors.

View Source 
var Tracef = func(format string, a ...interface{}) {}

Tracef is used for low-level tracing.

Functions

func Checksum 

func Checksum(bo binary.ByteOrder, s0, s1 uint32, b []byte) (uint32, uint32)

Checksum computes a running SQLite checksum over a byte slice.

func FormatWALPath added in v0.2.0 

func FormatWALPath(index int) string

FormatWALPath formats a WAL filename with a given index.

func FormatWALPathWithOffset added in v0.2.0 

func FormatWALPathWithOffset(index int, offset int64) string

FormatWALPathWithOffset formats a WAL filename with a given index & offset.

func IsGenerationName 

func IsGenerationName(s string) bool

IsGenerationName returns true if s is the correct length and is only lowercase hex characters.

func IsSnapshotPath 

func IsSnapshotPath(s string) bool

IsSnapshotPath returns true if s is a path to a snapshot file.

func IsWALPath 

func IsWALPath(s string) bool

IsWALPath returns true if s is a path to a WAL file.

func ParseSnapshotPath 

func ParseSnapshotPath(s string) (index int, ext string, err error)

ParseSnapshotPath returns the index for the snapshot. Returns an error if the path is not a valid snapshot path.

func ParseWALPath 

func ParseWALPath(s string) (index int, offset int64, ext string, err error)

ParseWALPath returns the index & offset for the WAL file. Returns an error if the path is not a valid snapshot path.

func SnapshotIndexAt added in v0.3.0 

func SnapshotIndexAt(ctx context.Context, r Replica, generation string, timestamp time.Time) (int, error)

SnapshotIndexAt returns the highest index for a snapshot within a generation that occurs before timestamp. If timestamp is zero, returns the latest snapshot.

func ValidateReplica added in v0.3.0 

func ValidateReplica(ctx context.Context, r Replica) error

ValidateReplica restores the most recent data from a replica and validates that the resulting database matches the current database.

func WALIndexAt added in v0.3.0 

func WALIndexAt(ctx context.Context, r Replica, generation string, maxIndex int, timestamp time.Time) (int, error)

WALIndexAt returns the highest index for a WAL file that occurs before maxIndex & timestamp. If timestamp is zero, returns the highest WAL index.

Types

type DB 

type DB struct {

	// Minimum threshold of WAL size, in pages, before a passive checkpoint.
	// A passive checkpoint will attempt a checkpoint but fail if there are
	// active transactions occurring at the same time.
	MinCheckpointPageN int

	// Maximum threshold of WAL size, in pages, before a forced checkpoint.
	// A forced checkpoint will block new transactions and wait for existing
	// transactions to finish before issuing a checkpoint and resetting the WAL.
	//
	// If zero, no checkpoints are forced. This can cause the WAL to grow
	// unbounded if there are always read transactions occurring.
	MaxCheckpointPageN int

	// Time between automatic checkpoints in the WAL. This is done to allow
	// more fine-grained WAL files so that restores can be performed with
	// better precision.
	CheckpointInterval time.Duration

	// Frequency at which to perform db sync.
	MonitorInterval time.Duration

	// List of replicas for the database.
	// Must be set before calling Open().
	Replicas []Replica
	// contains filtered or unexported fields
}

DB represents a managed instance of a SQLite database in the file system.

func NewDB 

func NewDB(path string) *DB

NewDB returns a new instance of DB for a given path.

func (*DB) CRC64 added in v0.3.0 

func (db *DB) CRC64() (uint64, Pos, error)

CRC64 returns a CRC-64 ISO checksum of the database and its current position.

This function obtains a read lock so it prevents syncs from occurring until the operation is complete. The database will still be usable but it will be unable to checkpoint during this time.

func (*DB) Checkpoint added in v0.3.0 

func (db *DB) Checkpoint(mode string) (err error)

Checkpoint performs a checkpoint on the WAL file.

func (*DB) Close 

func (db *DB) Close() (err error)

Close releases the read lock & closes the database. This method should only be called by tests as it causes the underlying database to be checkpointed.

func (*DB) CurrentGeneration 

func (db *DB) CurrentGeneration() (string, error)

CurrentGeneration returns the name of the generation saved to the "generation" file in the meta data directory. Returns empty string if none exists.

func (*DB) CurrentShadowWALIndex 

func (db *DB) CurrentShadowWALIndex(generation string) (index int, size int64, err error)

CurrentShadowWALIndex returns the current WAL index & total size.

func (*DB) CurrentShadowWALPath 

func (db *DB) CurrentShadowWALPath(generation string) (string, error)

CurrentShadowWALPath returns the path to the last shadow WAL in a generation.

func (*DB) GenerationNamePath 

func (db *DB) GenerationNamePath() string

GenerationNamePath returns the path of the name of the current generation.

func (*DB) GenerationPath 

func (db *DB) GenerationPath(generation string) string

GenerationPath returns the path of a single generation. Panics if generation is blank.

func (*DB) MetaPath 

func (db *DB) MetaPath() string

MetaPath returns the path to the database metadata.

func (*DB) Notify 

func (db *DB) Notify() <-chan struct{}

Notify returns a channel that closes when the shadow WAL changes.

func (*DB) Open 

func (db *DB) Open() (err error)

Open initializes the background monitoring goroutine.

func (*DB) PageSize 

func (db *DB) PageSize() int

PageSize returns the page size of the underlying database. Only valid after database exists & Init() has successfully run.

func (*DB) Path 

func (db *DB) Path() string

Path returns the path to the database.

func (*DB) Pos added in v0.2.0 

func (db *DB) Pos() (Pos, error)

Pos returns the current position of the database.

func (*DB) Replica added in v0.2.0 

func (db *DB) Replica(name string) Replica

Replica returns a replica by name.

func (*DB) Restore 

func (db *DB) Restore(ctx context.Context, opt RestoreOptions) error

Restore restores the database from a replica based on the options given. This method will restore into opt.OutputPath, if specified, or into the DB's original database path. It can optionally restore from a specific replica or generation or it will automatically choose the best one. Finally, a timestamp can be specified to restore the database to a specific point-in-time.

func (*DB) SQLDB added in v0.3.0 

func (db *DB) SQLDB() *sql.DB

SQLDB returns a reference to the underlying sql.DB connection.

func (*DB) ShadowWALDir added in v0.2.0 

func (db *DB) ShadowWALDir(generation string) string

ShadowWALDir returns the path of the shadow wal directory. Panics if generation is blank.

func (*DB) ShadowWALPath 

func (db *DB) ShadowWALPath(generation string, index int) string

ShadowWALPath returns the path of a single shadow WAL file. Panics if generation is blank or index is negative.

func (*DB) ShadowWALReader 

func (db *DB) ShadowWALReader(pos Pos) (r *ShadowWALReader, err error)

ShadowWALReader opens a reader for a shadow WAL file at a given position. If the reader is at the end of the file, it attempts to return the next file.

The caller should check Pos() & Size() on the returned reader to check offset.

func (*DB) Snapshots added in v0.2.0 

func (db *DB) Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

Snapshots returns a list of all snapshots across all replicas.

func (*DB) SoftClose 

func (db *DB) SoftClose() (err error)

SoftClose closes everything but the underlying db connection. This method is available because the binary needs to avoid closing the database on exit to prevent autocheckpointing.

func (*DB) Sync 

func (db *DB) Sync() (err error)

Sync copies pending data from the WAL to the shadow WAL.

func (*DB) UpdatedAt 

func (db *DB) UpdatedAt() (time.Time, error)

UpdatedAt returns the last modified time of the database or WAL file.

func (*DB) WALPath 

func (db *DB) WALPath() string

WALPath returns the path to the database's WAL file.

func (*DB) WALs added in v0.2.0 

func (db *DB) WALs(ctx context.Context) ([]*WALInfo, error)

WALs returns a list of all WAL files across all replicas.

type FileReplica 

type FileReplica struct {

	// Time to keep snapshots and related WAL files.
	// Database is snapshotted after interval and older WAL files are discarded.
	Retention time.Duration

	// Time between checks for retention.
	RetentionCheckInterval time.Duration

	// Time between validation checks.
	ValidationInterval time.Duration

	// If true, replica monitors database for changes automatically.
	// Set to false if replica is being used synchronously (such as in tests).
	MonitorEnabled bool
	// contains filtered or unexported fields
}

FileReplica is a replica that replicates a DB to a local file path.

func NewFileReplica 

func NewFileReplica(db *DB, name, dst string) *FileReplica

NewFileReplica returns a new instance of FileReplica.

func (*FileReplica) CalcPos added in v0.2.0 

func (r *FileReplica) CalcPos(ctx context.Context, generation string) (pos Pos, err error)

CalcPos returns the position for the replica for the current generation. Returns a zero value if there is no active generation.

func (*FileReplica) DB added in v0.3.0 

func (r *FileReplica) DB() *DB

DB returns the parent database reference.

func (*FileReplica) EnforceRetention added in v0.3.0 

func (r *FileReplica) EnforceRetention(ctx context.Context) (err error)

EnforceRetention forces a new snapshot once the retention interval has passed. Older snapshots and WAL files are then removed.

func (*FileReplica) GenerationDir added in v0.3.0 

func (r *FileReplica) GenerationDir(generation string) string

GenerationDir returns the path to a generation's root directory.

func (*FileReplica) GenerationStats 

func (r *FileReplica) GenerationStats(ctx context.Context, generation string) (stats GenerationStats, err error)

GenerationStats returns stats for a generation.

func (*FileReplica) Generations 

func (r *FileReplica) Generations(ctx context.Context) ([]string, error)

Generations returns a list of available generation names.

func (*FileReplica) LastPos added in v0.2.0 

func (r *FileReplica) LastPos() Pos

LastPos returns the last successfully replicated position.

func (*FileReplica) MaxSnapshotIndex added in v0.2.0 

func (r *FileReplica) MaxSnapshotIndex(generation string) (int, error)

MaxSnapshotIndex returns the highest index for the snapshots.

func (*FileReplica) Name 

func (r *FileReplica) Name() string

Name returns the name of the replica. Returns the type if no name set.

func (*FileReplica) Path added in v0.3.0 

func (r *FileReplica) Path() string

Path returns the path the replica was initialized with.

func (*FileReplica) SnapshotDir 

func (r *FileReplica) SnapshotDir(generation string) string

SnapshotDir returns the path to a generation's snapshot directory.

func (*FileReplica) SnapshotPath 

func (r *FileReplica) SnapshotPath(generation string, index int) string

SnapshotPath returns the path to a snapshot file.

func (*FileReplica) SnapshotReader 

func (r *FileReplica) SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

SnapshotReader returns a reader for snapshot data at the given generation/index. Returns os.ErrNotExist if no matching index is found.

func (*FileReplica) Snapshots added in v0.2.0 

func (r *FileReplica) Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

Snapshots returns a list of available snapshots in the replica.

func (*FileReplica) Start 

func (r *FileReplica) Start(ctx context.Context)

Start starts replication for a given generation.

func (*FileReplica) Stop 

func (r *FileReplica) Stop()

Stop cancels any outstanding replication and blocks until finished.

func (*FileReplica) Sync added in v0.2.0 

func (r *FileReplica) Sync(ctx context.Context) (err error)

Sync replays data from the shadow WAL into the file replica.

func (*FileReplica) Type 

func (r *FileReplica) Type() string

Type returns the type of replica.

func (*FileReplica) WALDir 

func (r *FileReplica) WALDir(generation string) string

WALDir returns the path to a generation's WAL directory

func (*FileReplica) WALPath 

func (r *FileReplica) WALPath(generation string, index int) string

WALPath returns the path to a WAL file.

func (*FileReplica) WALReader 

func (r *FileReplica) WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

WALReader returns a reader for WAL data at the given index. Returns os.ErrNotExist if no matching index is found.

func (*FileReplica) WALs added in v0.2.0 

func (r *FileReplica) WALs(ctx context.Context) ([]*WALInfo, error)

WALs returns a list of available WAL files in the replica.

type GenerationStats 

type GenerationStats struct {
	// Count of snapshot & WAL files.
	SnapshotN int
	WALN      int

	// Time range for the earliest snapshot & latest WAL file update.
	CreatedAt time.Time
	UpdatedAt time.Time
}

GenerationStats represents high level stats for a single generation.

type Pos 

type Pos struct {
	Generation string // generation name
	Index      int    // wal file index
	Offset     int64  // offset within wal file
}

Pos is a position in the WAL for a generation.

func (Pos) IsZero 

func (p Pos) IsZero() bool

IsZero returns true if p is the zero value.

func (Pos) String 

func (p Pos) String() string

String returns a string representation.

type Replica 

type Replica interface {
	// The name of the replica. Defaults to type if no name specified.
	Name() string

	// String identifier for the type of replica ("file", "s3", etc).
	Type() string

	// The parent database.
	DB() *DB

	// Starts replicating in a background goroutine.
	Start(ctx context.Context)

	// Stops all replication processing. Blocks until processing stopped.
	Stop()

	// Returns the last replication position.
	LastPos() Pos

	// Returns the computed position of the replica for a given generation.
	CalcPos(ctx context.Context, generation string) (Pos, error)

	// Returns a list of generation names for the replica.
	Generations(ctx context.Context) ([]string, error)

	// Returns basic information about a generation including the number of
	// snapshot & WAL files as well as the time range covered.
	GenerationStats(ctx context.Context, generation string) (GenerationStats, error)

	// Returns a list of available snapshots in the replica.
	Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

	// Returns a list of available WAL files in the replica.
	WALs(ctx context.Context) ([]*WALInfo, error)

	// Returns a reader for snapshot data at the given generation/index.
	SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

	// Returns a reader for WAL data at the given position.
	WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)
}

Replica represents a remote destination to replicate the database & WAL.

type RestoreOptions 

type RestoreOptions struct {
	// Target path to restore into.
	// If blank, the original DB path is used.
	OutputPath string

	// Specific replica to restore from.
	// If blank, all replicas are considered.
	ReplicaName string

	// Specific generation to restore from.
	// If blank, all generations considered.
	Generation string

	// Specific index to restore from.
	// Set to math.MaxInt64 to ignore index.
	Index int

	// Point-in-time to restore database.
	// If zero, database restore to most recent state available.
	Timestamp time.Time

	// If true, no actual restore is performed.
	// Only equivalent log output for a regular restore.
	DryRun bool

	// Logger used to print status to.
	Logger *log.Logger
}

RestoreOptions represents options for DB.Restore().

func NewRestoreOptions added in v0.2.0 

func NewRestoreOptions() RestoreOptions

NewRestoreOptions returns a new instance of RestoreOptions with defaults.

type ShadowWALReader 

type ShadowWALReader struct {
	// contains filtered or unexported fields
}

ShadowWALReader represents a reader for a shadow WAL file that tracks WAL position.

func (*ShadowWALReader) Close 

func (r *ShadowWALReader) Close() error

Close closes the underlying WAL file handle.

func (*ShadowWALReader) N 

func (r *ShadowWALReader) N() int64

N returns the remaining bytes in the reader.

func (*ShadowWALReader) Pos 

func (r *ShadowWALReader) Pos() Pos

Pos returns the current WAL position.

func (*ShadowWALReader) Read 

func (r *ShadowWALReader) Read(p []byte) (n int, err error)

Read reads bytes into p, updates the position, and returns the bytes read. Returns io.EOF at the end of the available section of the WAL.

type SnapshotInfo added in v0.2.0 

type SnapshotInfo struct {
	Name       string
	Replica    string
	Generation string
	Index      int
	Size       int64
	CreatedAt  time.Time
}

SnapshotInfo represents file information about a snapshot.

func FilterSnapshotsAfter added in v0.3.0 

func FilterSnapshotsAfter(a []*SnapshotInfo, t time.Time) []*SnapshotInfo

FilterSnapshotsAfter returns all snapshots that were created on or after t.

func FindMinSnapshotByGeneration added in v0.3.0 

func FindMinSnapshotByGeneration(a []*SnapshotInfo, generation string) *SnapshotInfo

FindMinSnapshotByGeneration finds the snapshot with the lowest index in a generation.

type WALInfo added in v0.2.0 

type WALInfo struct {
	Name       string
	Replica    string
	Generation string
	Index      int
	Offset     int64
	Size       int64
	CreatedAt  time.Time
}

WALInfo represents file information about a WAL file.

Source Files

View all Source files

Directories

Path Synopsis
cmd
litestream command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.