snapshot

README

Snapshots

Snapshots are Scoot's representation of file system states. Snapshots are immutable, and are labelled by a unique ID.

The snapshot code consists of the main Snapshot interface, and various implementations for dealing with snapshots in different filesystem paradigms - simple directories, git repositories, etc.

Key objects (not exhaustive, see code or godoc for more):

• Snapshots and Files
  • Snapshot - the main interface for representing snapshots and the files in them
  • File - interface representing a file open for reading
  • FileCursor - interface for reading specific segments of files
• Higher-level abstractions for filesystem data - implementations in snapshot/snaphots/
  • DB - interface to deal with Snapshots. Should replace the types below.
  • Filer - interface for treating snapshots as files, includes Checkouter and Ingester
  • Checkout - Specific instance or "checkout" of a snapshot
  • Checkouter - Wrapping interface for managing snapshots on the local filesystem
  • Ingester - interface for creating snapshots from the local filesystem
• git specific objects
  • package gitfiler - for checking out and handling snapshots from Git
    • RepoPool - for handling concurrent access to Git repos
    • RepoIniter - interface for controlling (possibly expensive) Git repo initialization
    • Checkouter - a Git-specific snapshot checkouter implementation
  • package gitdb - implementation of DB interface that stores local Snapshots in a git ODB

Documentation

Overview

package snapshot offers access to Snapshot.

The main entry point is the interface DB. DB holds Snapshots which can be an FSSnapshot or a GitCommitSnapshot.

An FSSnapshot is a snapshot of filesystem state (like a tar file).

A GitCommitSnapshot mirrors a git commit: a Snapshot, commit metadata, and an optional list of parent GitCommitSnapshots.

Snapshots can be used as the input to a Task, and we will store the output of a Task as a Snapshot.

There may be more Snapshot kinds in the future. E.g., a file (without the directory structure).

Package snapshot provides interfaces and implementations for Scoot snapshots, which represent immutable filesystem state. This includes concepts for Files, and various sim/test implementations.

Index

• Constants
• Variables
• type Checkout
• type Checkouter
• type Creator
• type DB
• type DefaultNoSuchSnapshotError
  • func (err *DefaultNoSuchSnapshotError) Errno() syscall.Errno
  • func (err *DefaultNoSuchSnapshotError) Error() string
  • func (err *DefaultNoSuchSnapshotError) NoSuchSnapshotError()
• type DefaultPathError
  • func (err *DefaultPathError) Errno() syscall.Errno
  • func (err *DefaultPathError) Error() string
  • func (err *DefaultPathError) PathError()
• type Dirent
• type File
• type FileCursor
  • func MakeCursor(f File) FileCursor
• type FileInfo
• type FileType
• type Filer
  • func NewDBAdapter(db DB) Filer
• type ID
• type Ingester
• type NoSuchSnapshotError
• type PathError
• type Reader
• type Snapshot
  • func NewBlacklistSnapshot(delegate Snapshot, blacklist map[string]bool) Snapshot
  • func NewFileBackedSnapshot(root string, id string) Snapshot
• type Snapshots
  • func NewBlacklistSnapshots(delegate Snapshots, blacklist map[string]bool) Snapshots
  • func NewFileBackedSnapshots(root string) Snapshots

Constants

const (
	// We only support the types we want to support. Anything else will be Unknown.
	FT_Unknown   FileType = 0
	FT_Directory          = 4
	FT_File               = 8
	FT_Symlink            = 12
)

Variables

var Trace bool

Types

type Checkout

type Checkout interface {
	// Path in the local filesystem to the Checkout
	Path() string

	// ID of the checked-out Snapshot
	ID() string

	// Releases this Checkout, allowing the Checkouter to clean/recycle this checkout.
	// After Release(), the client may not look at files under Path().
	Release() error
}

Checkout represents one checkout of a Snapshot. A Checkout is a copy of a Snapshot that lives in the local filesystem at a path.

type Checkouter

type Checkouter interface {
	// Checkout checks out the Snapshot identified by id, or an error if it fails.
	Checkout(id string) (Checkout, error)

	// Create checkout in a caller controlled dir.
	CheckoutAt(id string, dir string) (Checkout, error)
}

Checkouter allows reading a Snapshot into the local filesystem.

type Creator

type Creator interface {

	// IngestDir ingests a directory directly.
	// Creates an FSSnapshot whose contents are the same as the directory in the
	// local filesystem at the path identified by dir.
	// TODO(dbentley): define behavior on non-{file,directory} filetypes encountered
	// in dir, e.g. block devices or symlinks
	IngestDir(dir string) (ID, error)

	// IngestGitCommit ingests the commit identified by commitish from ingestRepo
	// commitish may be any string that identifies a commit
	// Creates a GitCommitSnapshot that mirrors the ingested commit.
	IngestGitCommit(ingestRepo *repo.Repository, commitish string) (ID, error)
}

Creator allows creating new Snapshots.

type DB

type DB interface {
	Creator
	Reader
}

DB is the full read-write Snapshot Database, allowing creation and reading of Snapshots.

type DefaultNoSuchSnapshotError

type DefaultNoSuchSnapshotError struct{}

func (*DefaultNoSuchSnapshotError) Errno

func (err *DefaultNoSuchSnapshotError) Errno() syscall.Errno

func (*DefaultNoSuchSnapshotError) Error

func (err *DefaultNoSuchSnapshotError) Error() string

func (*DefaultNoSuchSnapshotError) NoSuchSnapshotError

func (err *DefaultNoSuchSnapshotError) NoSuchSnapshotError()

type DefaultPathError

type DefaultPathError struct{}

func (*DefaultPathError) Errno

func (err *DefaultPathError) Errno() syscall.Errno

func (*DefaultPathError) Error

func (err *DefaultPathError) Error() string

func (*DefaultPathError) PathError

func (err *DefaultPathError) PathError()

type Dirent

type Dirent struct {
	Name string
	Type FileType
}

Dirent represents a file as stored in a directory. This is less info than is stored in a file, and so has less info than a stat returns.

type File

type File interface {
	// ReadAt reads len(b) bytes from the File starting at byte offset off.
	// It returns the number of bytes read and the error, if any.
	// ReadAt always returns a non-nil error when n < len(b).
	// At end of file, that error is io.EOF.
	ReadAt(p []byte, off int64) (int, error)

	// Read entire file, returning error for anything other than a complete read.
	// We define this rather than rely on ioutil.ReadAll to avoid coalescing small chunked reads.
	ReadAll() ([]byte, error)

	// Close closes the file, rendering it unusable for I/O.
	Close() error
}

File represents a file open for reading. An implementation of Snapshots may not keep a file open.

type FileCursor

type FileCursor interface {
	// Read reads up to len(b) bytes from the File. It returns the number of
	// bytes read and an error, if any. EOF is signaled by a zero count
	// with err set to io.EOF.
	// Read requires per-file state; an implementation is (for now, during
	// development) allowed to panic on a call to Read if it does not wish
	// to maintain that state.
	Read(p []byte) (n int, err error)

	// ReadAt reads len(b) bytes from the File starting at byte offset off.
	// It returns the number of bytes read and the error, if any.
	// ReadAt always returns a non-nil error when n < len(b).
	// At end of file, that error is io.EOF.
	ReadAt(p []byte, off int64) (n int, err error)

	// Close closes the file, rendering it unusable for I/O.
	Close() error
}

func MakeCursor

func MakeCursor(f File) FileCursor

TODO(dbentley): change to have File support a .ToCursor method.

type FileInfo

type FileInfo interface {
	Type() FileType
	IsExec() bool
	Size() int64
	IsDir() bool
}

FileInfo represents the stat of a file in a Snapshot. It has less info that os.FileInfo.

type FileType

type FileType int

type Filer

type Filer interface {
	Checkouter
	Ingester
}

A Filer lets clients deal with Snapshots as files in the local filesystem.

func NewDBAdapter

func NewDBAdapter(db DB) Filer

TODO: this is temporary until we finalize snapshot.DB and gitDB.

type ID

type ID string

ID identifies a Snapshot in DB. (Cf. doc.go for explanation of Snapshot) Opaque to the client.

type Ingester

type Ingester interface {
	// Takes an absolute path on the local filesystem.
	// The contents of path will be stored in a snapshot which may then be checked out by id.
	Ingest(path string) (id string, err error)

	// Takes a mapping of source paths to be copied into corresponding destination directories.
	// Source paths are absolute, and destination directories are relative to Checkout root.
	IngestMap(srcToDest map[string]string) (id string, err error)
}

Ingester creates a Snapshot from a path in the local filesystem.

type NoSuchSnapshotError

type NoSuchSnapshotError interface {
	NoSuchSnapshotError()
	Error() string
}

type PathError

type PathError interface {
	PathError()
	Error() string
}

TODO(dbentley): os returns PathError for many things in ways that are unspecified, so it's hard for us to be more exact, because os might change underneath us.

type Reader

type Reader interface {
	// ReadFileAll reads the contents of the file path in FSSnapshot ID, or errors
	ReadFileAll(id ID, path string) ([]byte, error)

	// Checkout puts the Snapshot identified by id in the local filesystem, returning
	// the path where it lives or an error.
	// TODO(dbentley): should we have separate methods based on the kind of Snapshot?
	Checkout(id ID) (path string, err error)

	// ReleaseCheckout releases a path from a previous Checkout. This allows Scoot to reuse
	// the path. Scoot will not touch path after Checkout until ReleaseCheckout.
	ReleaseCheckout(path string) error

	// ExportGitCommit puts the GitCommitSnapshot identified by id into exportRepo,
	// returning the sha of the exported commit.
	ExportGitCommit(id ID, exportRepo *repo.Repository) (commit string, err error)
}

Reader allows reading data from existing Snapshots

type Snapshot

type Snapshot interface {
	// The identifier of this Snapshot
	// Should be opaque to the client (they are handed IDs from somewhere and hand them to Snapshots).
	// Implementations might have IDs like git-<sha1 from git> or dbentley-newfeatures-12345
	// TODO(dbentley): namespaces for different implementations
	Id() string

	// Lstat returns a FileInfo describing the named file in the snapshot.
	// If the file is a symbolic link, the FileInfo describes the symbolic link.
	// LStat makes no attempt to follow the link. If there is an error because
	// the file doesn't exist it will be a PathError
	Lstat(name string) (FileInfo, error)

	// Stat returns a FileInfo describing the named file in the snapshot.
	// If the file is a symbolic link, the FileInfo describes destination
	// of the link. If there is an error because
	// the file doesn't exist in the snapshot, it will be a PathError
	Stat(name string) (FileInfo, error)

	// Readdirents reads the dirents of the named directory. If there is an error,
	// it will be a PathError.
	Readdirents(name string) ([]Dirent, error)

	// Readlink reads the destination of the named symbolic link.
	// If there is an error, it will be a PathError
	// If the named file is not a symbolic link, it will be a PathError
	Readlink(name string) (string, error)

	// Opens the named file. This is similar to os.Open, but an implementation
	// is allowed to defer checking if the file exists until the first
	// operation. If the file doesn't exist and the implementation checks,
	// the error will be a PathError.
	Open(path string) (File, error)
}

A read-only, immutable Snapshot of filesystem state

func NewBlacklistSnapshot

func NewBlacklistSnapshot(delegate Snapshot, blacklist map[string]bool) Snapshot

Creates a new Blacklisting Snapshot that delegates to delegate but blacklists keys in blacklist

func NewFileBackedSnapshot

func NewFileBackedSnapshot(root string, id string) Snapshot

type Snapshots

type Snapshots interface {
	// Get the Snapshot identified by id. If no such snapshot exists,
	// the error will be a NoSuchSnapshotError.
	Get(id string) (Snapshot, error)
}

func NewBlacklistSnapshots

func NewBlacklistSnapshots(delegate Snapshots, blacklist map[string]bool) Snapshots

Creates a new Blacklisting Snapshots that delegates to delegate but blacklists keys in blacklist

func NewFileBackedSnapshots

func NewFileBackedSnapshots(root string) Snapshots