core

package
v1.999.0-test Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2021 License: MIT Imports: 29 Imported by: 0

Documentation

Overview

Package core exposes core objects used by datamon: repos, bundles and labels.

Index

Constants

View Source 
const (
	// DiffEntryTypeAdd indicates the bundle exhibits an extra entry
	DiffEntryTypeAdd = iota
	// DiffEntryTypeDel indicates the bundle exhibits a missing entry
	DiffEntryTypeDel
	// DiffEntryTypeDif indicates the bundle exhibits different entries
	DiffEntryTypeDif
)

Variables

This section is empty.

Functions

func CreateDiamond 

func CreateDiamond(repo string, stores context2.Stores, opts ...DiamondOption) (model.DiamondDescriptor, error)

CreateDiamond persists an initialized diamond with a repo descriptor

func CreateRepo 

func CreateRepo(repo model.RepoDescriptor, stores context2.Stores) error

CreateRepo persists a repository with a repo descriptor and some context's stores

func CreateSplit 

func CreateSplit(repo, diamondID string, stores context2.Stores, opts ...SplitOption) (model.SplitDescriptor, error)

CreateSplit persists a new split for some initialized diamond for a repo

func DeleteBundle added in v1.1.0 

func DeleteBundle(repo string, stores context2.Stores, bundleID string) error

DeleteBundle removes a single bundle from a repo

func DeleteEntriesFromRepo added in v1.1.0 

func DeleteEntriesFromRepo(repo string, stores context2.Stores, toDelete []string) error

DeleteEntriesFromRepo remove a list of file entries from all bundles in a repo

func DeleteLabel added in v1.1.0 

func DeleteLabel(repo string, stores context2.Stores, name string) error

DeleteLabel removes a single label from a repo

func DeleteRepo added in v1.1.0 

func DeleteRepo(repo string, stores context2.Stores) error

DeleteRepo removes a repository from metadata

func DiamondExists 

func DiamondExists(repo, diamondID string, stores context2.Stores) error

DiamondExists checks if a diamond exists on a repo

func DownloadMetadata 

func DownloadMetadata(ctx context.Context, bundle *Bundle) (err error)

DownloadMetadata from the archive to main memory

func GetBundleStore 

func GetBundleStore(stores context2.Stores) storage.Store

GetBundleStore extracts the metadata store for bundles from some context's stores

func GetDiamond 

func GetDiamond(repo, diamondID string, stores context2.Stores, opts ...DiamondOption) (model.DiamondDescriptor, error)

GetDiamond retrieves a diamond

func GetDiamondStore 

func GetDiamondStore(stores context2.Stores) storage.Store

GetDiamondStore selects the metadata store for diamonds from a context

In the current setup, diamond metadata are located in the vmetadata store.

func GetLabelStore 

func GetLabelStore(stores context2.Stores) storage.Store

GetLabelStore tells which store holds label metadata

func GetLatestBundle 

func GetLatestBundle(repo string, stores context2.Stores) (string, error)

GetLatestBundle returns the latest bundle descriptor from a repo

func GetRepo 

func GetRepo(repo string, stores context2.Stores) (*model.RepoDescriptor, error)

GetRepo retrieves a repository

func GetRepoDescriptorByRepoName 

func GetRepoDescriptorByRepoName(stores context2.Stores, repoName string) (model.RepoDescriptor, error)

GetRepoDescriptorByRepoName returns the descriptor of a named repo

func GetRepoStore 

func GetRepoStore(stores context.Stores) storage.Store

GetRepoStore extracts the metadata store from some context's stores

NOTE: this is redundant with GetBundleStore

func GetSplit 

func GetSplit(repo, diamondID, splitID string, stores context2.Stores, opts ...SplitOption) (model.SplitDescriptor, error)

GetSplit retrieves a split

func GetSplitStore 

func GetSplitStore(stores context2.Stores) storage.Store

GetSplitStore selects the metadata store for splits from a context

In the current setup, split metadata are located in the vmetadata store.

func ListBundles 

func ListBundles(repo string, stores context2.Stores, opts ...Option) (model.BundleDescriptors, error)

ListBundles returns a list of bundle descriptors from a repo. It collects all bundles until completion.

NOTE: this func could become deprecated. At this moment, however, it is used by pkg/web.

func ListBundlesApply 

func ListBundlesApply(repo string, stores context2.Stores, apply ApplyBundleFunc, opts ...Option) error

ListBundlesApply applies some function to the retrieved bundles, in lexicographic order of keys.

The execution of the applied function does not block background retrieval of more keys and bundle descriptors.

Example usage: printing bundle descriptors as they come 

  err := core.ListBundlesApply(repo, store, func(bundle model.BundleDescriptor) error {
				fmt.Fprintf(os.Stderr, "%v\n", bundle)
				return nil
			})

func ListContexts 

func ListContexts(config storage.Store) ([]string, error)

ListContexts provides the list of available contexts in a remote configuration store, sorted.

func ListDiamonds 

func ListDiamonds(repo string, stores context2.Stores, opts ...Option) (model.DiamondDescriptors, error)

ListDiamonds yields all ongoing diamonds on a repo

func ListDiamondsApply 

func ListDiamondsApply(repo string, stores context2.Stores, apply ApplyDiamondFunc, opts ...Option) error

ListDiamondsApply applies some function to the retrieved diamonds, ordered by completion time.

The execution of the applied function does not block background retrieval of more keys and diamond descriptors.

Example usage: printing diamond descriptors as they come 

  err := core.ListDiamondsApply(repo, store, func(diamond model.DiamondDescriptor) error {
				fmt.Fprintf(os.Stderr, "%v\n", diamond)
				return nil
			})

func ListLabels 

func ListLabels(repo string, stores context2.Stores, opts ...Option) ([]model.LabelDescriptor, error)

ListLabels returns all labels from a repo

func ListLabelsApply 

func ListLabelsApply(repo string, store context2.Stores, apply ApplyLabelFunc, opts ...Option) error

ListLabelsApply applies some function to the retrieved labels, in lexicographic order of keys.

func ListRepos 

func ListRepos(stores context2.Stores, opts ...Option) ([]model.RepoDescriptor, error)

ListRepos returns all repos from a store

func ListReposApply 

func ListReposApply(stores context2.Stores, apply ApplyRepoFunc, opts ...Option) error

ListReposApply applies some function to the retrieved repos, in lexicographic order of keys.

func ListSplits 

func ListSplits(repo, diamondID string, stores context2.Stores, opts ...Option) (model.SplitDescriptors, error)

ListSplits yields all ongoing splits on a repo and a given diamond

func ListSplitsApply 

func ListSplitsApply(repo, diamondID string, stores context2.Stores, apply ApplySplitFunc, opts ...Option) error

ListSplitsApply applies some function to the retrieved splits, ordered by completion time.

The execution of the applied function does not block background retrieval of more keys and split descriptors.

Example usage: printing split descriptors as they come 

  err := core.ListSplitsApply(repo, store, func(split model.SplitDescriptor) error {
				fmt.Fprintf(os.Stderr, "%v\n", split)
				return nil
			})

func PopulateFiles 

func PopulateFiles(ctx context.Context, bundle *Bundle) (err error)

PopulateFiles populates a ConsumableStore with the metadata for this bundle

func Publish 

func Publish(ctx context.Context, bundle *Bundle) (err error)

Publish a bundle to a consumable store

func PublishFile 

func PublishFile(ctx context.Context, bundle *Bundle, file string) (err error)

PublishFile publish a single bundle file to a ConsumableStore

func PublishMetadata 

func PublishMetadata(ctx context.Context, bundle *Bundle) (err error)

PublishMetadata from the archive to the consumable store

func PublishSelectBundleEntries 

func PublishSelectBundleEntries(ctx context.Context, bundle *Bundle, selectionPredicate func(string) (bool, error)) (err error)

PublishSelectBundleEntries publish a selected list of entries from a bundle to a ConsumableStore, based on a predicate filter

func RenameRepo 

func RenameRepo(repo, newRepo string, stores context2.Stores) error

RenameRepo renames a repository in metadata

func RepoExists 

func RepoExists(repo string, stores context2.Stores) error

RepoExists checks for the existence of a repository by name in storage

func Update 

func Update(ctx context.Context, bundleSrc, bundleDest *Bundle) (err error)

Update a destination bundle from a source bundle

func Upload 

func Upload(ctx context.Context, bundle *Bundle, opts ...Option) (err error)

Upload an bundle to archive

func UploadSpecificKeys 

func UploadSpecificKeys(ctx context.Context, bundle *Bundle, getKeys func() ([]string, error), opts ...Option) (err error)

UploadSpecificKeys uploads some specified keys (files) within a bundle's consumable store

Types

type ApplyBundleFunc 

type ApplyBundleFunc func(model.BundleDescriptor) error

ApplyBundleFunc is a function to be applied on a bundle

type ApplyDiamondFunc 

type ApplyDiamondFunc func(model.DiamondDescriptor) error

ApplyDiamondFunc is a function to be applied on a diamond

type ApplyLabelFunc 

type ApplyLabelFunc func(model.LabelDescriptor) error

ApplyLabelFunc is a function to be applied on a label

type ApplyRepoFunc 

type ApplyRepoFunc func(model.RepoDescriptor) error

ApplyRepoFunc is a function to be applied on a repo

type ApplySplitFunc 

type ApplySplitFunc func(model.SplitDescriptor) error

ApplySplitFunc is a function to be applied on a split

type Bundle 

type Bundle struct {
	RepoID          string
	BundleID        string
	ConsumableStore storage.Store

	BundleDescriptor model.BundleDescriptor
	BundleEntries    []model.BundleEntry

	SkipOnError bool // When uploading files
	Retry       bool // When uploading files

	metrics.Enable
	// contains filtered or unexported fields
}

Bundle represents a bundle in its archived state.

A bundle is a point in time read-only view of a rep:branch and is composed of individual files. Analogous to a commit in git.

func NewBundle 

func NewBundle(opts ...BundleOption) *Bundle

NewBundle creates a new bundle

func (*Bundle) BlobStore 

func (b *Bundle) BlobStore() storage.Store

BlobStore defines the blob storage (part of the context) for a bundle

func (*Bundle) Exists 

func (b *Bundle) Exists(ctx context.Context) (bool, error)

Exists checks for the existence of this bundle in the repository

func (*Bundle) GetBundleEntries 

func (b *Bundle) GetBundleEntries() []model.BundleEntry

GetBundleEntries retrieves all entries in a bundle

func (*Bundle) InitializeBundleID 

func (b *Bundle) InitializeBundleID() error

InitializeBundleID creates and sets a new bundle ID

func (*Bundle) MetaStore 

func (b *Bundle) MetaStore() storage.Store

MetaStore yields the metadata store for the current bundle context

func (*Bundle) ReadLogStore 

func (b *Bundle) ReadLogStore() storage.Store

ReadLogStore yields the Read Log storage for the bundle context

func (*Bundle) UploadBundleEntries 

func (b *Bundle) UploadBundleEntries(ctx context.Context) (err error)

UploadBundleEntries uploads the current list of entries for that bundle

func (*Bundle) VMetaStore 

func (b *Bundle) VMetaStore() storage.Store

VMetaStore yields the metadata store for the current bundle context

func (*Bundle) WALStore 

func (b *Bundle) WALStore() storage.Store

WALStore yields the Write Ahead Log storage for the bundle context

type BundleDiff 

type BundleDiff struct {
	Entries []DiffEntry
}

BundleDiff describes all differences between two bundle

func Diff 

func Diff(ctx context.Context, existing, additional *Bundle) (BundleDiff, error)

Diff shows the differences between two bundles

type BundleIOMetrics 

type BundleIOMetrics struct {
	metrics.IOMetrics
	FileCount  *stats.Int64Measure `metric:"fileCount" description:"number of files in a bundle" tags:"kind,operation"`
	IndexCount *stats.Int64Measure `metric:"indexCount" description:"number of metadata index files in a bundle" tags:"kind,operation"`
}

BundleIOMetrics extends IOMetrics to capture bundle file count and metadata index files count

func (*BundleIOMetrics) BundleFiles 

func (b *BundleIOMetrics) BundleFiles(files, indices int64, operation string)

BundleFiles recors metrics about files in a bundle

type BundleOption 

type BundleOption func(*Bundle)

BundleOption is a functor to build a bundle with some options

func BundleDescriptor 

func BundleDescriptor(r *model.BundleDescriptor) BundleOption

BundleDescriptor sets the descriptor for this bundle

func BundleID 

func BundleID(bID string) BundleOption

BundleID defines the ID for a bundle

func BundleWithMetrics 

func BundleWithMetrics(enabled bool) BundleOption

BundleWithMetrics toggles metrics on a core Bundle object

func BundleWithRetry 

func BundleWithRetry(enabled bool) BundleOption

BundleWithRetry toggles exponential backoff retry logic on upload of core Bundle object

func ConcurrentFileDownloads 

func ConcurrentFileDownloads(concurrentFileDownloads int) BundleOption

ConcurrentFileDownloads tunes the level of concurrency when downloading bundle files

func ConcurrentFileUploads 

func ConcurrentFileUploads(concurrentFileUploads int) BundleOption

ConcurrentFileUploads tunes the level of concurrency when uploading bundle files

func ConcurrentFilelistDownloads 

func ConcurrentFilelistDownloads(concurrentFilelistDownloads int) BundleOption

ConcurrentFilelistDownloads tunes the level of concurrency when retrieving the list of files in a bundle

func ConsumableStore 

func ConsumableStore(store storage.Store) BundleOption

ConsumableStore defines the consumable storage for a bundle

func ContextStores 

func ContextStores(cs context2.Stores) BundleOption

ContextStores defines the set of stores to build a context for a bundle

func Logger 

func Logger(l *zap.Logger) BundleOption

Logger injects a logging facility into bundle core operations

func Repo 

func Repo(r string) BundleOption

Repo defines the repo a bundle belongs to

func SkipMissing 

func SkipMissing(s bool) BundleOption

SkipMissing indicates that bundle retrieval errors should be ignored. Currently not implementated.

type Diamond 

type Diamond struct {
	*Bundle

	DiamondDescriptor model.DiamondDescriptor

	metrics.Enable
	// contains filtered or unexported fields
}

Diamond is a generalization of Bundle to support diamond operations

func NewDiamond 

func NewDiamond(repo string, stores context2.Stores, opts ...DiamondOption) *Diamond

NewDiamond builds a new diamond instance.

The default diamond gets populated with a random KSUID as diamondID.

Default diamond has cnflicts handling enabled.

func (*Diamond) Cancel 

func (d *Diamond) Cancel() (err error)

Cancel a diamond

func (*Diamond) Commit 

func (d *Diamond) Commit(opts ...Option) error

Commit a diamond

func (*Diamond) WithState 

func (d *Diamond) WithState(state model.DiamondState) *Diamond

WithState returns the diamond with its state modified and transition timestamp updated

type DiamondOption 

type DiamondOption func(*Diamond)

DiamondOption is a functor to build a diamond with some options.

Since Diamond extends bundle, it gets most of its options.

func DiamondBundleID 

func DiamondBundleID(bundleID string) DiamondOption

DiamondBundleID defines the ID for the bundle produced by the diamond

NOTE: this require the package to be built with the bundle_preserve tag to enable this feature

func DiamondConcurrentFileDownloads 

func DiamondConcurrentFileDownloads(concurrentFileDownloads int) DiamondOption

DiamondConcurrentFileDownloads tunes the level of concurrency when downloading diamond files

func DiamondConcurrentFileUploads 

func DiamondConcurrentFileUploads(concurrentFileUploads int) DiamondOption

DiamondConcurrentFileUploads tunes the level of concurrency when uploading diamond files

func DiamondConcurrentFilelistDownloads 

func DiamondConcurrentFilelistDownloads(concurrentFilelistDownloads int) DiamondOption

DiamondConcurrentFilelistDownloads tunes the level of concurrency when retrieving the list of files in a diamond

func DiamondContextStores 

func DiamondContextStores(cs context2.Stores) DiamondOption

DiamondContextStores defines the set of stores to build a context for a diamond

func DiamondDescriptor 

func DiamondDescriptor(desc *model.DiamondDescriptor) DiamondOption

DiamondDescriptor defines the descriptor of the diamond

func DiamondLogger 

func DiamondLogger(l *zap.Logger) DiamondOption

DiamondLogger injects a logging facility into diamond core operations

func DiamondMessage 

func DiamondMessage(m string) DiamondOption

DiamondMessage defines the message of the bundle descriptor created by the diamond

func DiamondRepo 

func DiamondRepo(r string) DiamondOption

DiamondRepo defines the repo a diamond belongs to

func DiamondWithMetrics 

func DiamondWithMetrics(enabled bool) DiamondOption

DiamondWithMetrics toggles metrics on a core Diamond object

type DiffEntry 

type DiffEntry struct {
	Type DiffEntryType
	// could use a method rather than storing Name in order to curb memory use
	Name       string
	Existing   model.BundleEntry
	Additional model.BundleEntry
}

DiffEntry describes a single point of difference between two bundles

type DiffEntryType 

type DiffEntryType uint

DiffEntryType qualifies the type of difference between two bundles

func (DiffEntryType) String 

func (det DiffEntryType) String() string

type KeyFilter 

type KeyFilter func(string) bool

KeyFilter is a function to filter the files to upload

type KeyIterator 

type KeyIterator func(string) ([]string, error)

KeyIterator retrieves all keys from a store

type Label 

type Label struct {
	Descriptor model.LabelDescriptor

	metrics.Enable
	// contains filtered or unexported fields
}

Label describes a bundle label.

A label is a name given to a bundle, analogous to tags in git. Examples: Latest, production.

func NewLabel 

func NewLabel(opts ...LabelOption) *Label

NewLabel builds a new label with a descriptor

func (*Label) DownloadDescriptor 

func (label *Label) DownloadDescriptor(ctx context.Context, bundle *Bundle, checkRepoExists bool) (err error)

DownloadDescriptor retrieves the label descriptor for a bundle

func (*Label) DownloadDescriptorVersions 

func (label *Label) DownloadDescriptorVersions(ctx context.Context, bundle *Bundle, checkRepoExists bool) (lds []model.LabelDescriptor, err error)

DownloadDescriptorVersions retrieves all versions of a abel descriptor for a bundle

func (*Label) UploadDescriptor 

func (label *Label) UploadDescriptor(ctx context.Context, bundle *Bundle) (err error)

UploadDescriptor persists the label descriptor for a bundle

type LabelOption 

type LabelOption func(*Label)

LabelOption is a functor to build labels

func LabelDescriptor 

func LabelDescriptor(r *model.LabelDescriptor) LabelOption

LabelDescriptor sets the descriptor for this label

func LabelWithMetrics 

func LabelWithMetrics(enabled bool) LabelOption

LabelWithMetrics toggles metrics for this label

func LabelWithVersion 

func LabelWithVersion(version string) LabelOption

LabelWithVersion indicates we want to retrieve a specific version of this label object

type M 

type M struct {
	Volume struct {
		Bundles metrics.FilesMetrics `group:"bundles" description:"metrics about datasets (bundles)"`
		IO      BundleIOMetrics      `group:"io" description:"metrics about bundle IO operations"`
	} `group:"volumetry" description:""`
	Usage metrics.UsageMetrics `group:"telemetry" description:"usage stats for the core package"`
}

M describes metrics for the core package

type Option 

type Option func(*Settings)

Option sets options for listing core objects

func BatchSize 

func BatchSize(batchSize int) Option

BatchSize sets the batch window to fetch core objects. It defaults to defaultBatchSize

func ConcurrentList 

func ConcurrentList(concurrentList int) Option

ConcurrentList sets the max level of concurrency to retrieve core objects. It defaults to 2 x #cpus.

func WithDoneChan 

func WithDoneChan(done chan struct{}) Option

WithDoneChan sets a signaling channel controlled by the caller to interrupt ongoing goroutines

func WithLabelPrefix 

func WithLabelPrefix(prefix string) Option

WithLabelPrefix is an option for ListLabelsApply, to filter on labels with some given prefix

func WithLabelVersions 

func WithLabelVersions(enabled bool) Option

WithLabelVersions makes ListLabels list all versions of labels (requires the vmedata bucket to enable versioning)

func WithMemProf 

func WithMemProf(memProfDir string) Option

WithMemProf enables profiling and sets the memory profile directory (defaults to the current working directory).

func WithMetrics 

func WithMetrics(enabled bool) Option

WithMetrics toggles metrics for the core package and its dependencies (e.g. cafs)

type Settings 

type Settings struct {
	metrics.Enable
	// contains filtered or unexported fields
}

Settings defines various settings for core features

type Split 

type Split struct {
	*Bundle

	SplitDescriptor model.SplitDescriptor
	DiamondID       string

	metrics.Enable
	// contains filtered or unexported fields
}

Split is a generalization of a Bundle, to support diamond operations

func NewSplit 

func NewSplit(repo, diamondID string, stores context2.Stores, opts ...SplitOption) *Split

NewSplit builds a split for core operations (get, ...) and makes Bundle capabilities available to a Split

func (*Split) Upload 

func (s *Split) Upload(opts ...Option) error

Upload a dataset as a split, without committing as a bundle

func (*Split) WithState 

func (s *Split) WithState(state model.SplitState) *Split

WithState returns the split with the state updated

type SplitOption 

type SplitOption func(*Split)

SplitOption defines an option for a Split

func SplitConcurrentFileUploads 

func SplitConcurrentFileUploads(concurrentFileUploads int) SplitOption

SplitConcurrentFileUploads tunes the level of concurrency when uploading files

func SplitConsumableStore 

func SplitConsumableStore(store storage.Store) SplitOption

SplitConsumableStore defines the consumable storage for a split

func SplitDescriptor 

func SplitDescriptor(d *model.SplitDescriptor) SplitOption

SplitDescriptor sets the model descriptor for a Split

func SplitKeyFilter 

func SplitKeyFilter(f KeyFilter) SplitOption

SplitKeyFilter defines a filter on the keys to be uploaded.

func SplitKeyIterator 

func SplitKeyIterator(iterator KeyIterator) SplitOption

SplitKeyIterator defines a custom key iterator function to upload files. KeyIterator may be used independently from KeyFilter.

func SplitLogger 

func SplitLogger(l *zap.Logger) SplitOption

SplitLogger sets a logger on the split object

func SplitMustExist 

func SplitMustExist(d bool) SplitOption

SplitMustExist sets a split which must already be existing on metadata storage when created.

func SplitSkipMissing 

func SplitSkipMissing(skip bool) SplitOption

SplitSkipMissing indicates that file retrieval errors should be ignored

func SplitWithMetrics 

func SplitWithMetrics(enabled bool) SplitOption

SplitWithMetrics toggles metrics on a core Split object

Source Files

View all Source files

Directories

Path Synopsis
Package mocks provides some testing utilities to test bundles.
 Package mocks provides some testing utilities to test bundles.
Package status exports errors produced by the core package.
 Package status exports errors produced by the core package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.