artifact

Documentation

Overview

Package artifact contains a solution for storing and fetching versioned blobs of data that are resolved on demand.

By default, a configuration directory called .artifact is searched for up the directory roots. It looks for a file called config.json in it. It defines where to store resolved files as well as where to fetch blobs from. In addition, when unspecified, versioned blobs are stored alongside the configuration file in tree.json which is a directory tree mapping files to content addresses. The system uses concepts from content-addressable storage in order to simplify storage and offer deduplication.

Index

• Constants
• Variables
• func IsErrArtifactNotFound(err error) bool
• func MustNewPath(to string) string
• func MustPath(to string) string
• func NewErrArtifactNotFoundHash(hash string) error
• func NewErrArtifactNotFoundPath(path string) error
• func NewPath(to string) (string, error)
• func Path(to string) (string, error)
• func TestSetupGlobalCache(t *testing.T) (string, func())
• type Cache
  • func GlobalCache() (Cache, error)
  • func NewCache(config *Config) (Cache, error)
• type Config
  • func LoadConfig() (*Config, error)
  • func LoadConfigFromFile(path string) (*Config, error)
  • func (c *Config) Lookup(path string) (*TreeNode, error)
  • func (c *Config) RemovePath(path string)
  • func (c *Config) StoreHash(nodeHash string, nodeSize int, path []string)
  • func (c *Config) UnmarshalJSON(data []byte) error
• type FileSystemStoreConfig
  • func (c *FileSystemStoreConfig) Type() StoreType
• type GoogleStorageStoreConfig
  • func (c *GoogleStorageStoreConfig) Type() StoreType
• type Status
• type Store
  • func NewStore(config StoreConfig) (Store, error)
• type StoreConfig
• type StoreType
• type TreeNode
  • func (tn *TreeNode) IsInternal() bool
  • func (tn *TreeNode) MarshalJSON() ([]byte, error)
  • func (tn *TreeNode) UnmarshalJSON(data []byte) error
• type TreeNodeExternal
• type TreeNodeTree

Constants

const (
	ConfigName = "config.json"
	TreeName   = "tree.json"
)

The artifact file names.

const (
	StoreTypeFileSystem    = StoreType("fs")
	StoreTypeGoogleStorage = StoreType("google_storage")
)

The set of known store types.

const DefaultSourcePullSizeLimitBytes = 1 << 22

DefaultSourcePullSizeLimitBytes is the limit where if a normal pull happens, a file larger than this size will not be pulled down from source unless pull with ignoring the limit is used.

Variables

var (
	// DotDir is the location that artifact uses for itself.
	DotDir = ".artifact"

	// DefaultCachePath is the default relative location to store all cached
	// files (by hash).
	DefaultCachePath = filepath.Join(DotDir, "cache")

	// DefaultRootPath is the default relative location to store artifacts pulled
	// from the tree.
	DefaultRootPath = filepath.Join(DotDir, "data")
)

var ErrConfigNotFound = errors.Errorf("%q not found on system", ConfigName)

ErrConfigNotFound is used when the configuration file cannot be found anywhere.

var Logger = golog.NewDevelopmentLogger("artifact")

Logger is the global logger to use for artifact.

Functions

func IsErrArtifactNotFound

func IsErrArtifactNotFound(err error) bool

IsErrArtifactNotFound returns if the given error is any kind of artifact not found error.

func MustNewPath

func MustNewPath(to string) string

MustNewPath works like NewPath but panics if any error occurs. Useful in tests.

func MustPath

func MustPath(to string) string

MustPath works like Path but panics if any error occurs. Useful in tests.

func NewErrArtifactNotFoundHash

func NewErrArtifactNotFoundHash(hash string) error

NewErrArtifactNotFoundHash returns an error for when an artifact is not found by its hash.

func NewErrArtifactNotFoundPath

func NewErrArtifactNotFoundPath(path string) error

NewErrArtifactNotFoundPath returns an error for when an artifact is not found by its path.

func NewPath

func NewPath(to string) (string, error)

NewPath returns the would be path to an artifact on the local file system.

func Path

func Path(to string) (string, error)

Path returns the local file system path to the given artifact path. It errors if it does not exist or cannot be ensured to exist.

func TestSetupGlobalCache

func TestSetupGlobalCache(t *testing.T) (string, func())

TestSetupGlobalCache usage implies this test package can *not* have any tests run in parallel without risking incorrect usage of the global cache. Be advised.

Types

type Cache

type Cache interface {
	Store

	// NewPath returns where a user visible asset would belong
	NewPath(to string) string

	// Ensure guarantees that a user visible path is populated with
	// the cached artifact. This can error if the path is unknown
	// or a failure happens retrieving it from underlying Store.
	Ensure(path string, ignoreLimit bool) (string, error)

	// Remove removes the given path from the tree and root if present
	// but not from the cache. Use clean with cache true to clear out
	// the cache.
	Remove(path string) error

	// Clean makes sure that the user visible assets reflect 1:1
	// the versioned assets in the tree.
	Clean() error

	// WriteThroughUser makes sure that the user visible assets not
	// yet versioned are added to the tree and "written through" to
	// any stores responsible for caching.
	WriteThroughUser() error

	// Status inspects the root and returns a git like status of what is to
	// be added.
	Status() (*Status, error)

	// Close must be called in order to clean up any in use resources.
	Close() error
}

A Cache is similar to a store in functionality but it also has capabilities to refer to artifacts by their designated path name and not by a hash. In addition, it understands how to update the underlying stores that the cache is based off of as well as the user visible, versioned assets.

func GlobalCache

func GlobalCache() (Cache, error)

GlobalCache returns a cache to be used globally based on an automatically discovered configuration file. If the configuration file cannot be found, a noop implementation of the cache is created. In the future this should possibly automatically create the configuration file.

func NewCache

func NewCache(config *Config) (Cache, error)

NewCache returns a new cache based on the given config. It ensures that directory structures and stores are accessible in advance.

type Config

type Config struct {
	// Cache is where the hashed files should live. If unset, it defaults
	// to DefaultCachePath.
	Cache string

	// Root is where the files represented by the tree should live. These
	// reflect files in the cache but is based on the current tree. If unset,
	// it defaults to a data directory within DefaultCachePath.
	Root string

	// SourceStore is the configuration for where artifacts are remotely pushed
	// to and pulled from.
	SourceStore StoreConfig

	// SourcePullSizeLimit is the limit where if a normal pull happens,
	// a file larger than this size will not be pulled down from source
	// unless pull with ignoring the limit is used. If unset, DefaultSourcePullSizeLimitBytes
	// is used.
	SourcePullSizeLimit int

	// Ignore is a list of simple file names to ignore when scanning through
	// the root.
	Ignore []string
	// contains filtered or unexported fields
}

A Config describes how artifact should function.

func LoadConfig

func LoadConfig() (*Config, error)

LoadConfig attempts to automatically load an artifact config by searching for the default configuration file upwards in the file system.

func LoadConfigFromFile

func LoadConfigFromFile(path string) (*Config, error)

LoadConfigFromFile loads a Config from the given path. It also searches for an adjacent tree file (not required to exist).

func (*Config) Lookup

func (c *Config) Lookup(path string) (*TreeNode, error)

Lookup looks an artifact up by its path and returns its associated node if it exists.

func (*Config) RemovePath

func (c *Config) RemovePath(path string)

RemovePath removes nodes that fall into the given path.

func (*Config) StoreHash

func (c *Config) StoreHash(nodeHash string, nodeSize int, path []string)

StoreHash associates a path to the given node hash. The path is able to overwrite any existing one, so this method can be destructive if an external node were to replace an internal one.

func (*Config) UnmarshalJSON

func (c *Config) UnmarshalJSON(data []byte) error

UnmarshalJSON unmarshals the config from JSON data.

type FileSystemStoreConfig

type FileSystemStoreConfig struct {
	Path string `json:"path"`
}

FileSystemStoreConfig is for configuring a local file system based Store.

func (*FileSystemStoreConfig) Type

func (c *FileSystemStoreConfig) Type() StoreType

Type returns that this is a file system Store.

type GoogleStorageStoreConfig

type GoogleStorageStoreConfig struct {
	Bucket string `json:"bucket"`
}

GoogleStorageStoreConfig is for configuring a Google based Store that has its credentials automatically looked up.

func (*GoogleStorageStoreConfig) Type

func (c *GoogleStorageStoreConfig) Type() StoreType

Type returns that this is a Google storage Store.

type Status

type Status struct {
	Modified []string
	Unstored []string
}

Status describes how the tree has changed. We don't show removed because a removal is a direct operation done on the tree that is immediately reflected in source control.

type Store

type Store interface {
	Contains(hash string) error
	Load(hash string) (io.ReadCloser, error)
	Store(hash string, r io.Reader) error
}

A Store is responsible for loading and storing artifacts by their hashes and content.

func NewStore

func NewStore(config StoreConfig) (Store, error)

NewStore returns a new store based on the given config. It errors if making the store fails or the underlying type has no constructor.

type StoreConfig

type StoreConfig interface {
	// Type returns the type of the Store.
	Type() StoreType
}

A StoreConfig describes how to configure and set up a Store.

type StoreType

type StoreType string

A StoreType identifies a specific type of Store.

type TreeNode

type TreeNode struct {
	// contains filtered or unexported fields
}

A TreeNode represents a node in an artifact tree. The tree is a hierarchy of artifacts that mimics a file system.

func (*TreeNode) IsInternal

func (tn *TreeNode) IsInternal() bool

IsInternal returns if this node is an internal node.

func (*TreeNode) MarshalJSON

func (tn *TreeNode) MarshalJSON() ([]byte, error)

MarshalJSON marshals the node out into JSON.

func (*TreeNode) UnmarshalJSON

func (tn *TreeNode) UnmarshalJSON(data []byte) error

UnmarshalJSON unmarshals JSON into a specific tree node that may be internal or external.

type TreeNodeExternal

type TreeNodeExternal struct {
	Hash string `json:"hash"`
	Size int    `json:"size"`
}

A TreeNodeExternal is an external node representing the location of an artifact identified by its content hash.

type TreeNodeTree

type TreeNodeTree map[string]*TreeNode

A TreeNodeTree is an internal node with mappings to other nodes.