content

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2024 License: Apache-2.0 Imports: 45 Imported by: 4

Documentation

Overview

Package content implements repository support for content-addressable storage.

Index

Constants

View Source 
const (
	DefaultMetadataCacheSweepAge = 24 * time.Hour
	DefaultDataCacheSweepAge     = 10 * time.Minute
	DefaultIndexCacheSweepAge    = 1 * time.Hour
)

constants below specify how long to prevent cache entries from expiring.

View Source 
const (
	PackBlobIDPrefixRegular blob.ID = "p"
	PackBlobIDPrefixSpecial blob.ID = "q"

	NoCompression compression.HeaderID = 0

	FormatLogModule = "kopia/format"

	DefaultIndexVersion = 2
)

Prefixes for pack blobs.

View Source 
const BlobIDPrefixSession blob.ID = "s"

BlobIDPrefixSession is the prefix for blob IDs indicating active sessions. Each blob ID will consist of {sessionID}.{suffix}.

Variables

View Source 
var EmptyID = index.EmptyID

EmptyID is an empty content ID.

View Source 
var ErrContentNotFound = errors.New("content not found")

ErrContentNotFound is returned when content is not found.

View Source 
var PackBlobIDPrefixes = []blob.ID{
	PackBlobIDPrefixRegular,
	PackBlobIDPrefixSpecial,
}

PackBlobIDPrefixes contains all possible prefixes for pack blobs.

Functions

func IDsToStrings added in v0.10.6 

func IDsToStrings(input []ID) []string

IDsToStrings converts the IDs to strings.

Types

type CachingOptions 

type CachingOptions struct {
	CacheDirectory              string          `json:"cacheDirectory,omitempty"`
	ContentCacheSizeBytes       int64           `json:"maxCacheSize,omitempty"`
	ContentCacheSizeLimitBytes  int64           `json:"contentCacheSizeLimitBytes,omitempty"`
	MetadataCacheSizeBytes      int64           `json:"maxMetadataCacheSize,omitempty"`
	MetadataCacheSizeLimitBytes int64           `json:"metadataCacheSizeLimitBytes,omitempty"`
	MaxListCacheDuration        DurationSeconds `json:"maxListCacheDuration,omitempty"`
	MinMetadataSweepAge         DurationSeconds `json:"minMetadataSweepAge,omitempty"`
	MinContentSweepAge          DurationSeconds `json:"minContentSweepAge,omitempty"`
	MinIndexSweepAge            DurationSeconds `json:"minIndexSweepAge,omitempty"`
	HMACSecret                  []byte          `json:"-"`
}

CachingOptions specifies configuration of local cache.

func (*CachingOptions) CacheSubdirOrEmpty added in v0.13.0 

func (c *CachingOptions) CacheSubdirOrEmpty(subdir string) string

CacheSubdirOrEmpty returns path to a cache subdirectory or empty string if cache is disabled.

func (*CachingOptions) CloneOrDefault added in v0.6.0 

func (c *CachingOptions) CloneOrDefault() *CachingOptions

CloneOrDefault returns a clone of the caching options or empty options for nil.

func (*CachingOptions) EffectiveMetadataCacheSizeBytes added in v0.14.0 

func (c *CachingOptions) EffectiveMetadataCacheSizeBytes() int64

EffectiveMetadataCacheSizeBytes returns the effective metadata cache size.

type DurationSeconds added in v0.9.7 

type DurationSeconds float64

DurationSeconds represents the duration in seconds.

func (DurationSeconds) DurationOrDefault added in v0.9.7 

func (s DurationSeconds) DurationOrDefault(def time.Duration) time.Duration

DurationOrDefault returns the duration or the provided default if not set or zero.

type ID 

type ID = index.ID

ID is an identifier of content in content-addressable storage.

func IDFromHash added in v0.11.0 

func IDFromHash(prefix IDPrefix, hash []byte) (ID, error)

IDFromHash creates and validates content ID from a prefix and hash.

func IDsFromStrings added in v0.10.6 

func IDsFromStrings(str []string) ([]ID, error)

IDsFromStrings converts strings to IDs.

func ParseID added in v0.11.0 

func ParseID(s string) (ID, error)

ParseID parses the provided string as content ID.

type IDPrefix added in v0.11.0 

type IDPrefix = index.IDPrefix

IDPrefix represents a content ID prefix (empty string or single character between 'g' and 'z').

type IDRange added in v0.6.0 

type IDRange = index.IDRange

IDRange represents a range of IDs.

type IndexBlobReader added in v0.8.0 

type IndexBlobReader interface {
	ListIndexBlobInfos(ctx context.Context) ([]indexblob.Metadata, time.Time, error)
}

IndexBlobReader provides an API for reading index blobs.

type Info 

type Info = index.Info

Info describes a single piece of content.

func ParseIndexBlob added in v0.9.0 

func ParseIndexBlob(blobID blob.ID, encrypted gather.Bytes, crypter blobcrypto.Crypter) ([]Info, error)

ParseIndexBlob loads entries in a given index blob and returns them.

type IterateCallback 

type IterateCallback func(Info) error

IterateCallback is the function type used as a callback during content iteration.

type IterateOptions 

type IterateOptions struct {
	Range          IDRange
	IncludeDeleted bool
	Parallel       int
}

IterateOptions contains the options used for iterating over content.

type IteratePackOptions 

type IteratePackOptions struct {
	IncludePacksWithOnlyDeletedContent bool
	IncludeContentInfos                bool
	Prefixes                           []blob.ID
}

IteratePackOptions are the options used to iterate over packs.

type IteratePacksCallback 

type IteratePacksCallback func(PackInfo) error

IteratePacksCallback is the function type used as callback during pack iteration.

type ManagerOptions added in v0.6.0 

type ManagerOptions struct {
	TimeNow                func() time.Time // Time provider
	DisableInternalLog     bool
	PermissiveCacheLoading bool
}

ManagerOptions are the optional parameters for manager creation.

func (*ManagerOptions) CloneOrDefault added in v0.8.0 

func (o *ManagerOptions) CloneOrDefault() *ManagerOptions

CloneOrDefault returns a clone of provided ManagerOptions or default empty struct if nil.

type PackInfo 

type PackInfo struct {
	PackID       blob.ID
	ContentCount int
	TotalSize    int64
	ContentInfos []Info
}

PackInfo contains the data for a pack.

type Reader added in v0.8.0 

type Reader interface {
	// returns true if the repository supports content compression.
	// this may be slightly stale if the repository recently
	// got upgraded, in which case it will return false which is safe.
	SupportsContentCompression() bool
	ContentFormat() format.Provider
	GetContent(ctx context.Context, id ID) ([]byte, error)
	ContentInfo(ctx context.Context, id ID) (Info, error)
	IterateContents(ctx context.Context, opts IterateOptions, callback IterateCallback) error
	IteratePacks(ctx context.Context, opts IteratePackOptions, callback IteratePacksCallback) error
	ListActiveSessions(ctx context.Context) (map[SessionID]*SessionInfo, error)
	EpochManager(ctx context.Context) (*epoch.Manager, bool, error)
}

Reader defines content read API.

type SessionID added in v0.8.0 

type SessionID string

SessionID represents identifier of a session.

func SessionIDFromBlobID added in v0.8.0 

func SessionIDFromBlobID(b blob.ID) SessionID

SessionIDFromBlobID returns session ID from a given blob ID or empty string if it's not a session blob ID.

type SessionInfo added in v0.8.0 

type SessionInfo struct {
	ID             SessionID `json:"id"`
	StartTime      time.Time `json:"startTime"`
	CheckpointTime time.Time `json:"checkpointTime"`
	User           string    `json:"username"`
	Host           string    `json:"hostname"`
}

SessionInfo describes a particular session and is persisted in Session blob.

type SessionOptions added in v0.8.0 

type SessionOptions struct {
	SessionUser string
	SessionHost string
	OnUpload    func(int64)
}

SessionOptions specifies session options.

type SharedManager added in v0.8.0 

type SharedManager struct {
	Stats *Stats
	// contains filtered or unexported fields
}

SharedManager is responsible for read-only access to committed data.

func NewSharedManager added in v0.8.0 

func NewSharedManager(ctx context.Context, st blob.Storage, prov format.Provider, caching *CachingOptions, opts *ManagerOptions, repoLogManager *repodiag.LogManager, mr *metrics.Registry) (*SharedManager, error)

NewSharedManager returns SharedManager that is used by SessionWriteManagers on top of a repository.

func (*SharedManager) AlsoLogToContentLog added in v0.9.5 

func (sm *SharedManager) AlsoLogToContentLog(ctx context.Context) context.Context

AlsoLogToContentLog wraps the provided content so that all logs are also sent to internal content log.

func (*SharedManager) CloseShared added in v0.13.0 

func (sm *SharedManager) CloseShared(ctx context.Context) error

CloseShared releases all resources in a shared manager.

func (*SharedManager) CompactIndexes added in v0.9.0 

func (sm *SharedManager) CompactIndexes(ctx context.Context, opt indexblob.CompactOptions) error

CompactIndexes performs compaction of index blobs ensuring that # of small index blobs is below opt.maxSmallBlobs.

func (*SharedManager) EpochManager added in v0.9.0 

func (sm *SharedManager) EpochManager(ctx context.Context) (*epoch.Manager, bool, error)

EpochManager returns the epoch manager.

func (*SharedManager) IndexBlobs added in v0.8.0 

func (sm *SharedManager) IndexBlobs(ctx context.Context, includeInactive bool) ([]indexblob.Metadata, error)

IndexBlobs returns the list of active index blobs.

func (*SharedManager) IndexReaderV0 added in v0.13.0 

func (sm *SharedManager) IndexReaderV0() IndexBlobReader

IndexReaderV0 return an index reader for reading V0 indexes.

func (*SharedManager) IndexReaderV1 added in v0.13.0 

func (sm *SharedManager) IndexReaderV1() IndexBlobReader

IndexReaderV1 return an index reader for reading V1 indexes.

func (*SharedManager) IsReadOnly added in v0.14.0 

func (sm *SharedManager) IsReadOnly() bool

IsReadOnly returns whether this instance of the SharedManager only supports reads or if it also supports mutations to the index.

func (*SharedManager) LoadIndexBlob added in v0.13.0 

func (sm *SharedManager) LoadIndexBlob(ctx context.Context, ibid blob.ID, d *gather.WriteBuffer) ([]Info, error)

LoadIndexBlob return index information loaded from the specified blob.

func (*SharedManager) PrepareUpgradeToIndexBlobManagerV1 added in v0.9.0 

func (sm *SharedManager) PrepareUpgradeToIndexBlobManagerV1(ctx context.Context) error

PrepareUpgradeToIndexBlobManagerV1 prepares the repository for migrating to IndexBlobManagerV1.

func (*SharedManager) Refresh added in v0.9.0 

func (sm *SharedManager) Refresh(ctx context.Context) error

Refresh reloads the committed content indexes.

type Stats 

type Stats struct {
	// contains filtered or unexported fields
}

Stats exposes statistics about content operation.

func (*Stats) DecryptedBytes 

func (s *Stats) DecryptedBytes() int64

DecryptedBytes returns the approximate total number of decrypted bytes.

func (*Stats) EncryptedBytes 

func (s *Stats) EncryptedBytes() int64

EncryptedBytes returns the approximate total number of decrypted bytes.

func (*Stats) HashedContent added in v0.6.0 

func (s *Stats) HashedContent() (count uint32, bytes int64)

HashedContent returns the approximate hashed content count and their total size in bytes.

func (*Stats) InvalidContents 

func (s *Stats) InvalidContents() uint32

InvalidContents returns the approximate count of invalid contents found.

func (*Stats) ReadContent added in v0.6.0 

func (s *Stats) ReadContent() (count uint32, bytes int64)

ReadContent returns the approximate read content count and their total size in bytes.

func (*Stats) Reset 

func (s *Stats) Reset()

Reset clears all content statistics.

func (*Stats) ValidContents 

func (s *Stats) ValidContents() uint32

ValidContents returns the approximate count of valid contents found.

func (*Stats) WrittenContent added in v0.6.0 

func (s *Stats) WrittenContent() (count uint32, bytes int64)

WrittenContent returns the approximate written content count and their total size in bytes.

type WriteManager added in v0.8.0 

type WriteManager struct {
	*SharedManager
	// contains filtered or unexported fields
}

WriteManager builds content-addressable storage with encryption, deduplication and packaging on top of BLOB store.

func NewManagerForTesting added in v0.9.0 

func NewManagerForTesting(ctx context.Context, st blob.Storage, f format.Provider, caching *CachingOptions, options *ManagerOptions) (*WriteManager, error)

NewManagerForTesting creates new content manager with given packing options and a formatter.

func NewWriteManager added in v0.8.0 

func NewWriteManager(ctx context.Context, sm *SharedManager, options SessionOptions, writeManagerID string) *WriteManager

NewWriteManager returns a session write manager.

func (*WriteManager) ContentFormat added in v0.8.0 

func (bm *WriteManager) ContentFormat() format.Provider

ContentFormat returns formatting options.

func (*WriteManager) ContentInfo added in v0.8.0 

func (bm *WriteManager) ContentInfo(ctx context.Context, contentID ID) (Info, error)

ContentInfo returns information about a single content.

func (*WriteManager) DeleteContent added in v0.8.0 

func (bm *WriteManager) DeleteContent(ctx context.Context, contentID ID) error

DeleteContent marks the given contentID as deleted.

NOTE: To avoid race conditions only contents that cannot be possibly re-created should ever be deleted. That means that contents of such contents should include some element of randomness or a contemporaneous timestamp that will never reappear.

func (*WriteManager) DisableIndexFlush added in v0.8.0 

func (bm *WriteManager) DisableIndexFlush(ctx context.Context)

DisableIndexFlush increments the counter preventing automatic index flushes.

func (*WriteManager) DisableIndexRefresh added in v0.9.0 

func (bm *WriteManager) DisableIndexRefresh()

DisableIndexRefresh disables index refresh for the remainder of this session.

func (*WriteManager) EnableIndexFlush added in v0.8.0 

func (bm *WriteManager) EnableIndexFlush(ctx context.Context)

EnableIndexFlush decrements the counter preventing automatic index flushes. The flushes will be re-enabled when the index drops to zero.

func (*WriteManager) Flush added in v0.8.0 

func (bm *WriteManager) Flush(ctx context.Context) error

Flush completes writing any pending packs and writes pack indexes to the underlying storage. Any pending writes completed before Flush() has started are guaranteed to be committed to the repository before Flush() returns.

func (*WriteManager) GetContent added in v0.8.0 

func (bm *WriteManager) GetContent(ctx context.Context, contentID ID) (v []byte, err error)

GetContent gets the contents of a given content. If the content is not found returns ErrContentNotFound.

func (*WriteManager) IterateContents added in v0.8.0 

func (bm *WriteManager) IterateContents(ctx context.Context, opts IterateOptions, callback IterateCallback) error

IterateContents invokes the provided callback for each content starting with a specified prefix and possibly including deleted items.

func (*WriteManager) IteratePacks added in v0.8.0 

func (bm *WriteManager) IteratePacks(ctx context.Context, options IteratePackOptions, callback IteratePacksCallback) error

IteratePacks invokes the provided callback for all pack blobs.

func (*WriteManager) IterateUnreferencedBlobs added in v0.8.0 

func (bm *WriteManager) IterateUnreferencedBlobs(ctx context.Context, blobPrefixes []blob.ID, parallellism int, callback func(blob.Metadata) error) error

IterateUnreferencedBlobs returns the list of unreferenced storage blobs.

func (*WriteManager) ListActiveSessions added in v0.8.0 

func (bm *WriteManager) ListActiveSessions(ctx context.Context) (map[SessionID]*SessionInfo, error)

ListActiveSessions returns a set of all active sessions in a given storage.

func (*WriteManager) MetadataCache added in v0.10.7 

func (bm *WriteManager) MetadataCache() cache.ContentCache

MetadataCache returns an instance of metadata cache.

func (*WriteManager) PrefetchContents added in v0.10.6 

func (bm *WriteManager) PrefetchContents(ctx context.Context, contentIDs []ID, hint string) []ID

PrefetchContents fetches the provided content IDs into the cache. Note that due to cache configuration, it's not guaranteed that all contents will actually be added to the cache.

func (*WriteManager) RecoverIndexFromPackBlob added in v0.8.0 

func (bm *WriteManager) RecoverIndexFromPackBlob(ctx context.Context, packFile blob.ID, packFileLength int64, commit bool) ([]Info, error)

RecoverIndexFromPackBlob attempts to recover index blob entries from a given pack file. Pack file length may be provided (if known) to reduce the number of bytes that are read from the storage.

func (*WriteManager) Revision added in v0.8.0 

func (bm *WriteManager) Revision() int64

Revision returns data revision number that changes on each write or refresh.

func (*WriteManager) RewriteContent added in v0.8.0 

func (bm *WriteManager) RewriteContent(ctx context.Context, contentID ID) error

RewriteContent causes reads and re-writes a given content using the most recent format. TODO(jkowalski): this will currently always re-encrypt and re-compress data, perhaps consider a pass-through mode that preserves encrypted/compressed bits.

func (*WriteManager) SupportsContentCompression added in v0.9.0 

func (bm *WriteManager) SupportsContentCompression() bool

SupportsContentCompression returns true if content manager supports content-compression.

func (*WriteManager) UndeleteContent added in v0.8.0 

func (bm *WriteManager) UndeleteContent(ctx context.Context, contentID ID) error

UndeleteContent rewrites the content with the given ID if the content exists and is mark deleted. If the content exists and is not marked deleted, this operation is a no-op.

func (*WriteManager) WriteContent added in v0.8.0 

func (bm *WriteManager) WriteContent(ctx context.Context, data gather.Bytes, prefix index.IDPrefix, comp compression.HeaderID) (ID, error)

WriteContent saves a given content of data to a pack group with a provided name and returns a contentID that's based on the contents of data written.

Source Files

View all Source files

Directories

Path Synopsis
Package index manages content indices.
 Package index manages content indices.
Package indexblob manages sets of active index blobs.
 Package indexblob manages sets of active index blobs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.