content

package
v0.3.0-rc5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2019 License: Apache-2.0 Imports: 38 Imported by: 4

Documentation

Overview

Package content implements repository support for content-addressable storage.

Index

Constants

View Source 
const (
	PackBlobIDPrefixRegular blob.ID = "p"
	PackBlobIDPrefixSpecial blob.ID = "q"
)
View Source 
const DefaultEncryption = "SALSA20-HMAC"

DefaultEncryption is the name of the default encryption algorithm.

View Source 
const DefaultHash = "BLAKE2B-256-128"

DefaultHash is the name of the default hash algorithm.

Variables

View Source 
var ErrContentNotFound = errors.New("content not found")

ErrContentNotFound is returned when content is not found.

View Source 
var PackBlobIDPrefixes = []blob.ID{
	PackBlobIDPrefixRegular,
	PackBlobIDPrefixSpecial,
}

PackBlobIDPrefixes contains all possible prefixes for pack blobs.

Functions

func CreateHashAndEncryptor 

func CreateHashAndEncryptor(f *FormattingOptions) (HashFunc, Encryptor, error)

func RegisterEncryption 

func RegisterEncryption(name string, newEncryptor EncryptorFactory)

RegisterEncryption registers new encryption algorithm.

func RegisterHash 

func RegisterHash(name string, newHashFunc HashFuncFactory)

RegisterHash registers a hash function with a given name.

func SupportedEncryptionAlgorithms 

func SupportedEncryptionAlgorithms() []string

func SupportedHashAlgorithms 

func SupportedHashAlgorithms() []string

func UsingContentCache 

func UsingContentCache(ctx context.Context, enabled bool) context.Context

UsingContentCache returns a derived context that causes content manager to use cache.

func UsingListCache 

func UsingListCache(ctx context.Context, enabled bool) context.Context

UsingListCache returns a derived context that causes content manager to use cache.

Types

type CachingOptions 

type CachingOptions struct {
	CacheDirectory            string `json:"cacheDirectory,omitempty"`
	MaxCacheSizeBytes         int64  `json:"maxCacheSize,omitempty"`
	MaxMetadataCacheSizeBytes int64  `json:"maxMetadataCacheSize,omitempty"`
	MaxListCacheDurationSec   int    `json:"maxListCacheDuration,omitempty"`
	IgnoreListCache           bool   `json:"-"`
	HMACSecret                []byte `json:"-"`
}

CachingOptions specifies configuration of local cache.

type CompactOptions 

type CompactOptions struct {
	MinSmallBlobs        int
	MaxSmallBlobs        int
	AllIndexes           bool
	SkipDeletedOlderThan time.Duration
}

CompactOptions provides options for compaction

type Encryptor 

type Encryptor interface {
	// Encrypt returns encrypted bytes corresponding to the given plaintext.
	// Must not clobber the input slice and return ciphertext with additional padding and checksum.
	Encrypt(plainText, contentID []byte) ([]byte, error)

	// Decrypt returns unencrypted bytes corresponding to the given ciphertext.
	// Must not clobber the input slice. If IsAuthenticated() == true, Decrypt will perform
	// authenticity check before decrypting.
	Decrypt(cipherText, contentID []byte) ([]byte, error)

	// IsAuthenticated returns true if encryption is authenticated.
	// In this case Decrypt() is expected to perform authenticity check.
	IsAuthenticated() bool
}

Encryptor performs encryption and decryption of contents of data.

type EncryptorFactory 

type EncryptorFactory func(o *FormattingOptions) (Encryptor, error)

EncryptorFactory creates new Encryptor for given FormattingOptions

type Format 

type Format struct {
	Version    byte   // format version number must be 0x01
	KeySize    byte   // size of each key in bytes
	EntrySize  uint16 // size of each entry in bytes, big-endian
	EntryCount uint32 // number of sorted (key,value) entries that follow

	Entries []struct {
		Key   []byte // key bytes (KeySize)
		Entry entry
	}

	ExtraData []byte // extra data
}

Format describes a format of a single pack index. The actual structure is not used, it's purely for documentation purposes. The struct is byte-aligned.

type FormattingOptions 

type FormattingOptions struct {
	Version     int    `json:"version,omitempty"`     // version number, must be "1"
	Hash        string `json:"hash,omitempty"`        // identifier of the hash algorithm used
	Encryption  string `json:"encryption,omitempty"`  // identifier of the encryption algorithm used
	HMACSecret  []byte `json:"secret,omitempty"`      // HMAC secret used to generate encryption keys
	MasterKey   []byte `json:"masterKey,omitempty"`   // master encryption key (SIV-mode encryption only)
	MaxPackSize int    `json:"maxPackSize,omitempty"` // maximum size of a pack object
}

FormattingOptions describes the rules for formatting contents in repository.

func (*FormattingOptions) DeriveKey 

func (o *FormattingOptions) DeriveKey(purpose []byte, length int) []byte

DeriveKey uses HKDF to derive a key of a given length and a given purpose.

type HashFunc 

type HashFunc func(data []byte) []byte

HashFunc computes hash of content of data using a cryptographic hash function, possibly with HMAC and/or truncation.

type HashFuncFactory 

type HashFuncFactory func(o *FormattingOptions) (HashFunc, error)

HashFuncFactory returns a hash function for given formatting options.

type ID 

type ID string

ID is an identifier of content in content-addressable storage.

func (ID) HasPrefix 

func (i ID) HasPrefix() bool

HasPrefix determines if the given ID has a non-empty prefix.

func (ID) Prefix 

func (i ID) Prefix() ID

Prefix returns a one-character prefix of a content ID or an empty string.

type IndexBlobInfo 

type IndexBlobInfo struct {
	BlobID    blob.ID
	Length    int64
	Timestamp time.Time
}

IndexBlobInfo is an information about a single index blob managed by Manager.

type Info 

type Info struct {
	ID               ID      `json:"contentID"`
	Length           uint32  `json:"length"`
	TimestampSeconds int64   `json:"time"`
	PackBlobID       blob.ID `json:"packFile,omitempty"`
	PackOffset       uint32  `json:"packOffset,omitempty"`
	Deleted          bool    `json:"deleted"`
	Payload          []byte  `json:"payload"` // set for payloads stored inline
	FormatVersion    byte    `json:"formatVersion"`
}

Info is an information about a single piece of content managed by Manager.

func (*Info) Timestamp 

func (i *Info) Timestamp() time.Time

Timestamp returns the time when a content was created or deleted.

type IterateCallback 

type IterateCallback func(Info) error

type IterateOptions 

type IterateOptions struct {
	Prefix         ID
	IncludeDeleted bool
	Parallel       int
}

type IteratePackOptions 

type IteratePackOptions struct {
	IncludePacksWithOnlyDeletedContent bool
	IncludeContentInfos                bool
}

type IteratePacksCallback 

type IteratePacksCallback func(PackInfo) error

type Manager 

type Manager struct {
	Format         FormattingOptions
	CachingOptions CachingOptions
	// contains filtered or unexported fields
}

Manager builds content-addressable storage with encryption, deduplication and packaging on top of BLOB store.

func NewManager 

func NewManager(ctx context.Context, st blob.Storage, f *FormattingOptions, caching CachingOptions, repositoryFormatBytes []byte) (*Manager, error)

NewManager creates new content manager with given packing options and a formatter.

func (*Manager) Close 

func (bm *Manager) Close()

Close closes the content manager.

func (*Manager) CompactIndexes 

func (bm *Manager) CompactIndexes(ctx context.Context, opt CompactOptions) error

CompactIndexes performs compaction of index blobs ensuring that # of small contents is between minSmallContentCount and maxSmallContentCount

func (*Manager) ContentInfo 

func (bm *Manager) ContentInfo(ctx context.Context, contentID ID) (Info, error)

ContentInfo returns information about a single content.

func (*Manager) DeleteContent 

func (bm *Manager) DeleteContent(contentID ID) error

DeleteContent marks the given contentID as deleted.

NOTE: To avoid race conditions only contents that cannot be possibly re-created should ever be deleted. That means that contents of such contents should include some element of randomness or a contemporaneous timestamp that will never reappear.

func (*Manager) DisableIndexFlush 

func (bm *Manager) DisableIndexFlush()

DisableIndexFlush increments the counter preventing automatic index flushes.

func (*Manager) EnableIndexFlush 

func (bm *Manager) EnableIndexFlush()

EnableIndexFlush decrements the counter preventing automatic index flushes. The flushes will be reenabled when the index drops to zero.

func (*Manager) Flush 

func (bm *Manager) Flush(ctx context.Context) error

Flush completes writing any pending packs and writes pack indexes to the underlyign storage.

func (*Manager) GetContent 

func (bm *Manager) GetContent(ctx context.Context, contentID ID) ([]byte, error)

GetContent gets the contents of a given content. If the content is not found returns ErrContentNotFound.

func (*Manager) IndexBlobs 

func (bm *Manager) IndexBlobs(ctx context.Context) ([]IndexBlobInfo, error)

IndexBlobs returns the list of active index blobs.

func (*Manager) IterateContentInShortPacks 

func (bm *Manager) IterateContentInShortPacks(threshold int64, callback IterateCallback) error

IterateContentInShortPacks invokes the provided callback for all contents that are stored in packs shorter than the given threshold.

func (*Manager) IterateContents 

func (bm *Manager) IterateContents(opts IterateOptions, callback IterateCallback) error

IterateContents invokes the provided callback for each content starting with a specified prefix and possibly including deleted items.

func (*Manager) IteratePacks 

func (bm *Manager) IteratePacks(options IteratePackOptions, callback IteratePacksCallback) error

IteratePacks invokes the provided callback for all pack blobs.

func (*Manager) IterateUnreferencedBlobs 

func (bm *Manager) IterateUnreferencedBlobs(ctx context.Context, parallellism int, callback func(blob.Metadata) error) error

FindUnreferencedBlobs returns the list of unreferenced storage blobs.

func (*Manager) RecoverIndexFromPackBlob 

func (bm *Manager) RecoverIndexFromPackBlob(ctx context.Context, packFile blob.ID, packFileLength int64, commit bool) ([]Info, error)

RecoverIndexFromPackBlob attempts to recover index blob entries from a given pack file. Pack file length may be provided (if known) to reduce the number of bytes that are read from the storage.

func (*Manager) Refresh 

func (bm *Manager) Refresh(ctx context.Context) (bool, error)

Refresh reloads the committed content indexes.

func (*Manager) ResetStats 

func (bm *Manager) ResetStats()

ResetStats resets statistics to zero values.

func (*Manager) RewriteContent 

func (bm *Manager) RewriteContent(ctx context.Context, contentID ID) error

RewriteContent causes reads and re-writes a given content using the most recent format.

func (*Manager) Stats 

func (bm *Manager) Stats() Stats

Stats returns statistics about content manager operations.

func (*Manager) WriteContent 

func (bm *Manager) WriteContent(ctx context.Context, data []byte, prefix ID) (ID, error)

WriteContent saves a given content of data to a pack group with a provided name and returns a contentID that's based on the contents of data written.

type PackInfo 

type PackInfo struct {
	PackID       blob.ID
	ContentCount int
	TotalSize    int64
	ContentInfos []Info
}

type Stats 

type Stats struct {
	// Keep int64 fields first to ensure they get aligned to at least 64-bit boundaries
	// which is required for atomic access on ARM and x86-32.
	ReadBytes      int64 `json:"readBytes,omitempty"`
	WrittenBytes   int64 `json:"writtenBytes,omitempty"`
	DecryptedBytes int64 `json:"decryptedBytes,omitempty"`
	EncryptedBytes int64 `json:"encryptedBytes,omitempty"`
	HashedBytes    int64 `json:"hashedBytes,omitempty"`

	ReadContents    int32 `json:"readContents,omitempty"`
	WrittenContents int32 `json:"writtenContents,omitempty"`
	CheckedContents int32 `json:"checkedContents,omitempty"`
	HashedContents  int32 `json:"hashedContents,omitempty"`
	InvalidContents int32 `json:"invalidContents,omitempty"`
	PresentContents int32 `json:"presentContents,omitempty"`
	ValidContents   int32 `json:"validContents,omitempty"`
}

Stats exposes statistics about content operation.

func (*Stats) Reset 

func (s *Stats) Reset()

Reset clears all repository statistics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.