snapshot

README

Snapshots

Snapshots are Scoot's representation of file system states. Snapshots are immutable, and are labelled by a unique ID.

The snapshot code consists of the main Snapshot interface, and various implementations for dealing with snapshots in different filesystem paradigms - simple directories, git repositories, etc.

Key objects (not exhaustive, see code or godoc for more):

• Snapshots and Files
  • Snapshot - the main interface for representing snapshots and the files in them
  • File - interface representing a file open for reading
  • FileCursor - interface for reading specific segments of files
• Higher-level abstractions for filesystem data - implementations in snapshot/snaphots/
  • DB - interface to deal with Snapshots. Should replace the types below.
  • Filer - interface for treating snapshots as files, includes Checkouter, Ingester and Updater
  • Checkout - Specific instance or "checkout" of a snapshot
  • Checkouter - Wrapping interface for managing snapshots on the local filesystem
  • Ingester - interface for creating snapshots from the local filesystem
  • Updater - interface for updating underlying resource a Filer is utilizing
• git specific objects
  • RepoPool - for handling concurrent access to Git repos
  • RepoIniter - interface for controlling (possibly expensive) Git repo initialization
  • Checkouter - a Git-specific snapshot checkouter implementation
  • package gitdb - implementation of DB interface that stores local Snapshots in a git ODB

Snapshot Stores and Servers

Store

snapshot/store contains Snapshot storage interfaces and implementations.

Bundlestore

snapshot/bundlestore contains a server for storing and retrieving Snapshots from a store.

Documentation

Overview

package snapshot offers access to Snapshot.

The main entry point is the interface DB. DB holds Snapshots which can be an FSSnapshot or a GitCommitSnapshot.

An FSSnapshot is a snapshot of filesystem state (like a tar file).

A GitCommitSnapshot mirrors a git commit: a Snapshot, commit metadata, and an optional list of parent GitCommitSnapshots.

Snapshots can be used as the input to a Task, and we will store the output of a Task as a Snapshot.

There may be more Snapshot kinds in the future. E.g., a file (without the directory structure).

Index

• Constants
• func NewDBAdapter(db DB) *dbAdapter
• func NewNopCheckout(id, dir string) *nopCheckout
• type Checkout
• type Checkouter
• type Creator
• type DB
• type Filer
• type FilerAndInitDoneCh
• type ID
• type Ingester
• type InitDoneCh
• type Reader
• type Updater

Constants

const NoDuration time.Duration = time.Duration(0)

Functions

func NewDBAdapter

func NewDBAdapter(db DB) *dbAdapter

NewDBAdapter returns a *dbAdapter that implements the snapshot.Filer and snapshot.DB interfaces

func NewNopCheckout

func NewNopCheckout(id, dir string) *nopCheckout

NewNopCheckout returns a *nopCheckout that implements snapshot.Checkout with no-ops.

Types

type Checkout

type Checkout interface {
	// Path in the local filesystem to the Checkout
	Path() string

	// ID of the checked-out Snapshot
	ID() string

	// Releases this Checkout, allowing the Checkouter to clean/recycle this checkout.
	// After Release(), the client may not look at files under Path().
	Release() error
}

Checkout represents one checkout of a Snapshot. A Checkout is a copy of a Snapshot that lives in the local filesystem at a path.

type Checkouter

type Checkouter interface {
	// Checkout checks out the Snapshot identified by id, or an error if it fails.
	Checkout(id string) (Checkout, error)

	// Create checkout in a caller controlled dir.
	CheckoutAt(id string, dir string) (Checkout, error)

	// Request to cancel any current Checkouter operations
	CancelCheckout() error
}

Checkouter allows reading a Snapshot into the local filesystem.

type Creator

type Creator interface {

	// IngestDir ingests a directory directly.
	// Creates an FSSnapshot whose contents are the same as the directory in the
	// local filesystem at the path identified by dir.
	// TODO(dbentley): define behavior on non-{file,directory} filetypes encountered
	// in dir, e.g. block devices or symlinks
	IngestDir(dir string) (ID, error)

	// IngestGitCommit ingests the commit identified by commitish from ingestRepo
	// commitish may be any string that identifies a commit
	// Creates a GitCommitSnapshot that mirrors the ingested commit.
	IngestGitCommit(ingestRepo *repo.Repository, commitish string) (ID, error)

	// IngestGitWorkingDir ingests HEAD + working dir modifications from ingestRepo.
	// Creates a GitCommitSnapshot that mirrors the ingested commit.
	IngestGitWorkingDir(ingestRepo *repo.Repository) (ID, error)
}

Creator allows creating new Snapshots.

type DB

type DB interface {
	Creator
	Reader
	Updater
	Cancel() error // Request to cancel in-progress operations
}

TODO remove this abstraction, or consolidate it with Filer DB is the full read-write Snapshot Database, allowing creation and reading of Snapshots, and updating of the underlying DB resource.

type Filer

type Filer interface {
	Checkouter
	Ingester
	Updater
}

A Filer lets clients deal with Snapshots as files in the local filesystem.

type FilerAndInitDoneCh

type FilerAndInitDoneCh struct {
	Filer Filer
	IDC   InitDoneCh
}

Association of Filer and corresponding InitDoneCh, to support management of multiple Filers

type ID

type ID string

ID identifies a Snapshot in DB. (Cf. doc.go for explanation of Snapshot) Opaque to the client.

type Ingester

type Ingester interface {
	// Takes an absolute path on the local filesystem.
	// The contents of path will be stored in a snapshot which may then be checked out by id.
	Ingest(path string) (id string, err error)

	// Takes a mapping of source paths to be copied into corresponding destination directories.
	// Source paths are absolute, and destination directories are relative to Checkout root.
	IngestMap(srcToDest map[string]string) (id string, err error)

	// Request to cancel any current Ingester operations
	CancelIngest() error
}

Ingester creates a Snapshot from a path in the local filesystem.

type InitDoneCh

type InitDoneCh <-chan error

DB may require some form of initialization, in which case this chan should be provided by the db impl.

type Reader

type Reader interface {
	// ReadFileAll reads the contents of the file path in FSSnapshot ID, or errors
	ReadFileAll(id ID, path string) ([]byte, error)

	// Checkout puts the Snapshot identified by id in the local filesystem, returning
	// the path where it lives or an error.
	// TODO(dbentley): should we have separate methods based on the kind of Snapshot?
	Checkout(id ID) (path string, err error)

	// ReleaseCheckout releases a path from a previous Checkout. This allows Scoot to reuse
	// the path. Scoot will not touch path after Checkout until ReleaseCheckout.
	ReleaseCheckout(path string) error

	// ExportGitCommit puts the GitCommitSnapshot identified by id into exportRepo,
	// returning the sha of the exported commit.
	ExportGitCommit(id ID, exportRepo *repo.Repository) (commit string, err error)
}

Reader allows reading data from existing Snapshots

type Updater

type Updater interface {
	// Trigger an update on the underlying resource
	Update() error

	// Get the configured update frequency from the Updater.
	// This lets us use the high-level interface to control update concurrency.
	UpdateInterval() time.Duration
}

Updater allows Filers to have a means to manage updates on the underlying resources