model

package
v1.999.0-test Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2021 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package model describes the base objects manipulated by datamon.

The package exposes a model for metadata.

The object model for datamon is composed of: 

Repos:
  A datamon repository is analogous to a git repo. A repo is a dataset that has a unified lifecycle.
  A particular version of the files in a repo is called a bundle.

Bundles:
  A bundle is a point in time read-only view of a repo, composed of individual files.
  This is analogous to a commit in git.

Labels:
  A name given to a bundle, analogous to tags in git. Examples: Latest, production.

Contexts:
  A context provides a way to define multiple instances of datamon. Example: development, production

WAL:
  A write-ahead log tracks all changes to a repo.

Index

Constants

View Source 
const (
	// ConsumableStorePathTypeDescriptor defines consumable store metadata of type "descriptor"
	ConsumableStorePathTypeDescriptor byte = iota

	// ConsumableStorePathTypeFileList defines consumable store metadata of type "file list"
	ConsumableStorePathTypeFileList
)
View Source 
const (
	// ContextVersion indicates the version of the context model
	ContextVersion = 1.0
)
View Source 
const (
	// CurrentBundleVersion indicates the version of the bundle model
	//
	// Note that version numbering is an integer, not a semver string.
	//
	// TODO(fred): we should move to semver and use some semver comparison package whenever we want to compare versions.
	//
	// Change log from version 1:
	// - added support for diamond workflow (non breaking)
	CurrentBundleVersion uint64 = 2
)
View Source 
const TokenGeneratorPath = "WALTokenGeneratorPath"

TokenGeneratorPath is the path to the WAL token generator

View Source 
const (
	Version = 1
)

Variables

This section is empty.

Functions

func GenerateCheckpointPath 

func GenerateCheckpointPath(splitID, pth string) string

GenerateCheckpointPath builds a path in the dataset to save checkpointed files

func GenerateConflictPath 

func GenerateConflictPath(splitID, pth string) string

GenerateConflictPath builds a path in the dataset to save conflicting files

func GetArchivePathPrefixToBundles 

func GetArchivePathPrefixToBundles(repo string) string

GetArchivePathPrefixToBundles yields a path to all bundles in a repo.

Example: 

bundles/{repo}/

func GetArchivePathPrefixToContexts 

func GetArchivePathPrefixToContexts() string

GetArchivePathPrefixToContexts returns the path to the list of contexts

func GetArchivePathPrefixToDiamonds 

func GetArchivePathPrefixToDiamonds(repo string) string

GetArchivePathPrefixToDiamonds yields a path to all diamonds in a repo.

Example: 

diamonds/{repo}/

func GetArchivePathPrefixToLabels 

func GetArchivePathPrefixToLabels(repo string, prefixes ...string) string

GetArchivePathPrefixToLabels yields the path to labels in a repo, given some prefixes

func GetArchivePathPrefixToRepos 

func GetArchivePathPrefixToRepos() string

GetArchivePathPrefixToRepos yields the path to all repos

func GetArchivePathPrefixToSplits 

func GetArchivePathPrefixToSplits(repo, diamondID string) string

GetArchivePathPrefixToSplits yields a path to all splits in a diamond in a repo.

Example: 

diamonds/{repo}/{diamond}/

func GetArchivePathToBundle 

func GetArchivePathToBundle(repo string, bundleID string) string

GetArchivePathToBundle yields a path in a repo to the descriptor of a bundle

Example: 

bundles/{repo}/{bundleID}/bundle.yaml

func GetArchivePathToBundleFileList 

func GetArchivePathToBundleFileList(repo string, bundleID string, index uint64) string

GetArchivePathToBundleFileList yields a path to the list of the files in a bundle

Example: 

bundles/{repo}/{bundleID}/bundle-files-{index}.yaml

func GetArchivePathToDiamond 

func GetArchivePathToDiamond(repo, diamondID string, state DiamondState) string

GetArchivePathToDiamond yields a path in a repo to the descriptor of a diamond in any state.

Example: 

diamonds/{repo}/{diamond}/diamond-*.yaml

func GetArchivePathToFinalDiamond 

func GetArchivePathToFinalDiamond(repo, diamondID string) string

GetArchivePathToFinalDiamond yields a path in a repo to the descriptor of a diamond in a final state.

Example: 

diamonds/{repo}/{diamond}/diamond-done.yaml

func GetArchivePathToFinalSplit 

func GetArchivePathToFinalSplit(repo, diamondID, splitID string) string

GetArchivePathToFinalSplit yields a path in a repo to the descriptor of a split in a final state.

Example: 

diamonds/{repo}/{diamond}/splits/{split}/split-done.yaml

func GetArchivePathToInitialDiamond 

func GetArchivePathToInitialDiamond(repo, diamondID string) string

GetArchivePathToInitialDiamond yields a path in a repo to the descriptor of a diamond in an initialized state.

Example: 

diamonds/{repo}/{diamond}/diamond-running.yaml

func GetArchivePathToInitialSplit 

func GetArchivePathToInitialSplit(repo, diamondID, splitID string) string

GetArchivePathToInitialSplit yields a path in a repo to the descriptor of a split in a running state.

Example: 

diamonds/{repo}/{diamond}/splits/split}/split-running.yaml

func GetArchivePathToLabel 

func GetArchivePathToLabel(repo string, labelName string) string

GetArchivePathToLabel gets the path to the label descriptor.

Example: 

labels/{repo}/{label}/label.yaml

func GetArchivePathToRepoDescriptor 

func GetArchivePathToRepoDescriptor(repo string) string

GetArchivePathToRepoDescriptor yields a path in a repo to the descriptor of a bundle

Example: 

repos/{repo}/{bundleID}/repo.yaml

func GetArchivePathToSplit 

func GetArchivePathToSplit(repo, diamondID, splitID string, state SplitState) string

GetArchivePathToSplit yields a path in a repo to the descriptor of a split in any state.

Example: 

diamonds/{repo}/{diamond}/splits/{split}/split-*.yaml

func GetArchivePathToSplitFileList 

func GetArchivePathToSplitFileList(repo, diamondID, splitID string, generationID string, index uint64) string

GetArchivePathToSplitFileList yields a path to the list of the files in a diamond split

Example: 

diamonds/{repo}/{diamond}/splits/{split}/{generation}/bundle-files-{index}.yaml

func GetBlobName 

func GetBlobName(context string, n string) string

GetBlobName yields the name of the Blob store

func GetBundleTimeStamp 

func GetBundleTimeStamp() time.Time

GetBundleTimeStamp yields the current UTC time

func GetConsumablePathToBundle 

func GetConsumablePathToBundle(bundleID string) string

GetConsumablePathToBundle yields a path to some bundle metadata

func GetConsumablePathToBundleFileList 

func GetConsumablePathToBundleFileList(bundleID string, index uint64) string

GetConsumablePathToBundleFileList yields a path to some bundle file

func GetMetadataName 

func GetMetadataName(context string, n string) string

GetMetadataName yields the name of the Metadata store

func GetPathToCategory 

func GetPathToCategory(category string) string

GetPathToCategory returns the path to the descriptor for a Category

func GetPathToContainer 

func GetPathToContainer(category string, run string, runID string, stage string, stageID string, container string) string

GetPathToContainer returns the path to the descriptor Container for a container part of a stage in a run.

func GetPathToContext 

func GetPathToContext(context string) string

GetPathToContext returns the path to the context descriptor.

func GetPathToDataSetIn 

func GetPathToDataSetIn(category string, run string, runID string, stage string, stageID string, container string) string

GetPathToDataSetIn returns the path to the descriptor DataSetIn for a container part of a stage in a run.

func GetPathToDataSetOut 

func GetPathToDataSetOut(category string, run string, runID string, stage string, stageID string, container string) string

GetPathToDataSetOut returns the path to the descriptor DataSetOut for a container part of a stage in a run.

func GetPathToRun 

func GetPathToRun(category string, name string, id string) string

GetPathToRun returns the path to the descriptor for a Run

func GetPathToRunStatus 

func GetPathToRunStatus(category string, run string, runID string) string

GetPathToRunStatus returns the path to the descriptor RunStatus for a run.

func GetReadLogName 

func GetReadLogName(context string, n string) string

GetReadLogName yields the name of the Read Log store

func GetVMetadataName 

func GetVMetadataName(context string, n string) string

GetVMetadataName yields the name of the Versioning Metadata store

func GetWALName 

func GetWALName(context string, n string) string

GetWALName yields the name of the Write Ahead Log store

func IsGeneratedFile 

func IsGeneratedFile(file string) bool

IsGeneratedFile indicates if some file comes from auto-generation (e.g. .datamon files)

More generally, it indicates any special path on a consumable store not to be uploaded with a bundle.

func MarshalContext 

func MarshalContext(c *Context) ([]byte, error)

MarshalContext marshals a context as a YAML descriptor

func MarshalWAL 

func MarshalWAL(entry *Entry) ([]byte, error)

MarshalWAL marshals a WAL entry as a YAML descriptor

func ValidateContext 

func ValidateContext(context Context) error

ValidateContext checks the context is valid, i.e. all expected stores are well defined

func ValidateLabel 

func ValidateLabel(label LabelDescriptor) error

ValidateLabel validates a label descriptor

func ValidateRepo 

func ValidateRepo(repo RepoDescriptor) error

ValidateRepo validates a repository descriptor: name and description are required, name only contains letters, digits or '-'.

Types

type ArchivePathComponents 

type ArchivePathComponents struct {
	Repo            string
	BundleID        string
	ArchiveFileName string
	LabelName       string
	Context         string
	DiamondID       string
	SplitID         string
	GenerationID    string
	IsFinalState    bool
}

ArchivePathComponents defines the unique path parts to retrieve a file in a bundle

func GetArchivePathComponents 

func GetArchivePathComponents(archivePath string) (ArchivePathComponents, error)

GetArchivePathComponents yields all metadata components from a parsed archive path.

type BundleDescriptor 

type BundleDescriptor struct {
	LeafSize               uint32        `json:"leafSize" yaml:"leafSize"`                               // Bundles blobs are independently generated
	ID                     string        `json:"id" yaml:"id"`                                           // Unique ID for the bundle.
	Message                string        `json:"message" yaml:"message"`                                 // Message for the commit/bundle
	Parents                []string      `json:"parents,omitempty" yaml:"parents,omitempty"`             // Bundles with parent child relation
	Timestamp              time.Time     `json:"timestamp,omitempty" yaml:"timestamp,omitempty"`         // Local wall clock time
	Contributors           []Contributor `json:"contributors" yaml:"contributors"`                       // Contributor for the bundle
	BundleEntriesFileCount uint64        `json:"count" yaml:"count"`                                     // Number of file index files in this bundle
	Version                uint64        `json:"version,omitempty" yaml:"version,omitempty"`             // Version for the metadata model used for this bundle
	Deduplication          string        `json:"deduplication,omitempty" yaml:"deduplication,omitempty"` // Deduplication scheme used
	RunStage               string        `json:"runstage,omitempty" yaml:"runstage,omitempty"`           // Path to the run stage
	// contains filtered or unexported fields
}

BundleDescriptor represents a commit which is a file tree with the changes to the repository.

func NewBundleDescriptor 

func NewBundleDescriptor(opts ...BundleDescriptorOption) *BundleDescriptor

NewBundleDescriptor builds a new default bundle descriptor

type BundleDescriptorOption 

type BundleDescriptorOption func(descriptor *BundleDescriptor)

BundleDescriptorOption is a functor to build a bundle descriptor with some options

func BundleContributor 

func BundleContributor(c Contributor) BundleDescriptorOption

BundleContributor defines a single contributor for a bundle descriptor

func BundleContributors 

func BundleContributors(c []Contributor) BundleDescriptorOption

BundleContributors defines the list of contributors for a bundle descriptor

func Deduplication 

func Deduplication(d string) BundleDescriptorOption

Deduplication defines the deduplication scheme for a bundle descriptor

func Message 

func Message(m string) BundleDescriptorOption

Message defines the message of the bundle descriptor

func Parents 

func Parents(p []string) BundleDescriptorOption

Parents defines the parents for a bundle descriptor

type BundleDescriptors 

type BundleDescriptors []BundleDescriptor

BundleDescriptors is a sortable slice of BundleDescriptor

func (BundleDescriptors) Last 

func (b BundleDescriptors) Last() BundleDescriptor

Last bundle descriptor in slice

func (BundleDescriptors) Len 

func (b BundleDescriptors) Len() int

func (BundleDescriptors) Less 

func (b BundleDescriptors) Less(i, j int) bool

func (BundleDescriptors) Swap 

func (b BundleDescriptors) Swap(i, j int)

type BundleEntries 

type BundleEntries struct {
	BundleEntries []BundleEntry `json:"BundleEntries" yaml:"BundleEntries"`
	// contains filtered or unexported fields
}

BundleEntries are the files in this bundle.

type BundleEntry 

type BundleEntry struct {
	Hash         string      `json:"hash" yaml:"hash"`
	NameWithPath string      `json:"name" yaml:"name"`
	FileMode     os.FileMode `json:"mode" yaml:"mode"`
	Size         uint64      `json:"size" yaml:"size"`
	Timestamp    time.Time   `json:"timestamp,omitempty" yaml:"timestamp,omitempty"` // time the file was uploaded. Only serialized with entries uploaded by splits (not bundles)
	// contains filtered or unexported fields
}

BundleEntry describes a file in the bundle. Empty directories are skipped

type Category 

type Category struct {
	Name        string      `json:"name" yaml:"name"`               // Unique name for the category
	Contributor Contributor `json:"contributor" yaml:"contributor"` // Credentials for who created the category.
	Version     int         `json:"version" yaml:"version"`         // Version of the run Schema.
}

Category of run declares the kind of run it is.

type ConflictMode 

type ConflictMode string

ConflictMode indicates the conflict detection mode defined for a diamond 

const (
	// IgnoreConflicts is the diamond mode with which conflicts are not handled (latest win, no track kept of clobbered files)
	IgnoreConflicts ConflictMode = "ignored"

	// EnableCheckpoints is the diamond mode with which conflicts are explicitly handled and saved as "checkpoints" (incremental upload)
	EnableCheckpoints ConflictMode = "enable-checkpoints"

	// EnableConflicts is the diamond mode with which conflicts are detected and saved as "conflicts"
	EnableConflicts ConflictMode = "enable-conflicts"

	// ForbidConflicts is the diamond mode with which conflicts result in failure to commit
	ForbidConflicts ConflictMode = "forbids-conflicts"
)

type ConsumableStorePathMetadata 

type ConsumableStorePathMetadata struct {
	Type     byte
	BundleID string
	Index    uint64
}

ConsumableStorePathMetadata defines the metadata associated to a consumable store.

func GetConsumableStorePathMetadata 

func GetConsumableStorePathMetadata(path string) (ConsumableStorePathMetadata, error)

GetConsumableStorePathMetadata is the inverse of GetConsumablePath* functions.

The GetConsumablePath* functions return bundle.ConsumableStore keys (paths) given some parameters from the bundle.

This function, given one of the paths returned by a GetConsumablePath* function, parses the path and returns the input values to that function.

type ConsumableStorePathMetadataErr 

type ConsumableStorePathMetadataErr struct {
	// contains filtered or unexported fields
}

ConsumableStorePathMetadataErr defines errors related to consumable store metadata

func (ConsumableStorePathMetadataErr) Error 

func (e ConsumableStorePathMetadataErr) Error() string

type Container 

type Container struct {
	Image   string `json:"image" yaml:"image"`     // Image name
	ImageID string `json:"imageid" yaml:"imageid"` // Image ID
}

Container describes the details for a container in a stage.

type Context 

type Context struct {
	Name      string `json:"name" yaml:"name"`           // Name for the context
	WAL       string `json:"wal" yaml:"wal"`             // WAL is the location for the log
	ReadLog   string `json:"readlog" yaml:"readlog"`     // Read log is the location for read log.
	Blob      string `json:"blob" yaml:"blob"`           // Blob is the location for the data blobs
	Metadata  string `json:"metadata" yaml:"metadata"`   // Metadata is the location for the immutable metadata
	VMetadata string `json:"vmetadata" yaml:"vmetadata"` // VMetadata is the location for the mutable versioned metadata.
	Version   uint64 `json:"version" yaml:"version"`     // Version for the
	// contains filtered or unexported fields
}

Context defines the details for a datamon context.

func UnmarshalContext 

func UnmarshalContext(b []byte) (*Context, error)

UnmarshalContext unmarshals a context from a YAML descriptor

type Contributor 

type Contributor struct {
	Name  string `json:"name" yaml:"name"`
	Email string `json:"email" yaml:"email"`
	// contains filtered or unexported fields
}

Contributor who created the object

func (*Contributor) String 

func (c *Contributor) String() string

type DataSetIn 

type DataSetIn struct {
	Timestamp   time.Time `json:"timestamp" yaml:"timestamp"`     // Timestamp for the dataset in
	Stage       string    `json:"stage" yaml:"stage"`             // Stage in which the dataset is used
	StageID     string    `json:"stageid" yaml:"stageid"`         // StageID is a unique id for each instance of a stage.
	Container   string    `json:"container" yaml:"container"`     // Container which using the dataset
	MountPath   string    `json:"mountpath" yaml:"mountpath"`     // Path to mount it to
	Filesystems []FS      `json:"filesystems" yaml:"filesystems"` // Filesystem details
}

DataSetIn describes the details for the data set that went into a container.

type DataSetOut 

type DataSetOut struct {
	Timestamp   time.Time `json:"timestamp" yaml:"timestamp"`     // Timestamp for the dataset in
	Stage       string    `json:"stage" yaml:"stage"`             // Stage in which the dataset is used
	Container   string    `json:"container" yaml:"container"`     // Container which using the dataset
	MountPath   string    `json:"mountpath" yaml:"mountpath"`     // Path to mount it to
	Filesystems []FS      `json:"filesystems" yaml:"filesystems"` // Filesystem details
}

DataSetOut describes the details for the data set generated by a container.

type DiamondDescriptor 

type DiamondDescriptor struct {
	DiamondID      string            `json:"diamondID" yaml:"diamondID"`
	StartTime      time.Time         `json:"startTime" yaml:"startTime"`                           // documentary: when the diamond was initialized
	EndTime        time.Time         `json:"endTime,omitempty" yaml:"endTime,omitempty"`           // documentary: the diamond completion time (i.e. when the diamond reached a terminal state)
	State          DiamondState      `json:"state" yaml:"state"`                                   // the captured state of the diamond
	Mode           ConflictMode      `json:"mode" yaml:"mode"`                                     // the conflict handling mode defined at commit time. The default value is "enable-conflicts"
	HasConflicts   bool              `json:"hasConflicts,omitempty" yaml:"hasConflicts,omitempty"` // documentary snapshot of the outcome of conflict handling after commit
	HasCheckpoints bool              `json:"hasCheckpoints,omitempty" yaml:"hasCheckpoints,omitempty"`
	Tag            string            `json:"tag,omitempty" yaml:"tag,omitempty"`           // user-defined tag to taint logs
	BundleID       string            `json:"bundleID,omitempty" yaml:"bundleID,omitempty"` // keeps track of the produced bundle, when the diamond is successfully committed
	Splits         []SplitDescriptor `json:"splits,omitempty" yaml:"splits,omitempty"`     // documentary snapshot of the splits collected after a successful commit
	// contains filtered or unexported fields
}

DiamondDescriptor models a diamond's metadata

func NewDiamondDescriptor 

func NewDiamondDescriptor(opts ...DiamondDescriptorOption) *DiamondDescriptor

NewDiamondDescriptor builds a new DiamondDescriptor

type DiamondDescriptorOption 

type DiamondDescriptorOption func(*DiamondDescriptor)

DiamondDescriptorOption defines an option to build a DiamondDescriptor

func DiamondClone 

func DiamondClone(m DiamondDescriptor) DiamondDescriptorOption

DiamondClone clones from a DiamondDescriptor

func DiamondID 

func DiamondID(id string) DiamondDescriptorOption

DiamondID sets the DiamondID of a DiamondDescriptor

func DiamondMode 

func DiamondMode(mode ConflictMode) DiamondDescriptorOption

DiamondMode sets the conflict resolution mode for a diamond

func DiamondTag 

func DiamondTag(tag string) DiamondDescriptorOption

DiamondTag sets an informative tag on the diamond

type DiamondDescriptors 

type DiamondDescriptors []DiamondDescriptor

DiamondDescriptors is a sortable slice of DiamondDescriptor

func (DiamondDescriptors) Last 

func (b DiamondDescriptors) Last() DiamondDescriptor

Last bundle descriptor in slice

func (DiamondDescriptors) Len 

func (b DiamondDescriptors) Len() int

func (DiamondDescriptors) Less 

func (b DiamondDescriptors) Less(i, j int) bool

func (DiamondDescriptors) Swap 

func (b DiamondDescriptors) Swap(i, j int)

type DiamondState 

type DiamondState string

DiamondState models the running status of an ungoing diamond workflow 

const (
	// DiamondInitialized is the state of an initialized diamond
	DiamondInitialized DiamondState = "initialized"

	// DiamondDone indicates the diamond has completed with a successful commit. This is a terminal state.
	DiamondDone DiamondState = "done"

	// DiamondCanceled indicates the diamond has completed with a cancel. This is a terminal state.
	DiamondCanceled DiamondState = "canceled"
)

func (DiamondState) IsValid 

func (s DiamondState) IsValid() bool

IsValid checks the value of a diamond state

func (DiamondState) String 

func (s DiamondState) String() string

type Entry 

type Entry struct {
	Token   string `json:"token" yaml:"token"`
	Payload string `json:"payload" yaml:"payload"`
}

Entry defines a Write Ahead Log entry

func NewEntry 

func NewEntry(token string, payload string) *Entry

NewEntry creates a new entry for the WAL

func UnmarshalWAL 

func UnmarshalWAL(b []byte) (*Entry, error)

UnmarshalWAL unmarshals a WAL entry from a YAML descriptor

type FS 

type FS struct {
	Context    string `json:"context" yaml:"context"`       // Context for the dataset
	Repo       string `json:"repo" yaml:"repo"`             // Repo for the dataset
	BundleID   string `json:"bundleid" yaml:"bundleid"`     // BundleID to fetch
	Label      string `json:"label" yaml:"label"`           // Label to fetch
	Fuse       bool   `json:"fuse" yaml:"fuse"`             // Fuse mount for the dataset
	Stream     bool   `json:"stream" yaml:"stream"`         // Stream the data in
	VolumeName string `json:"volumename" yaml:"volumename"` // Volume name
}

FS captures the details for the filesystem mount of datasets.

type LabelDescriptor 

type LabelDescriptor struct {
	Name         string        `json:"name,omitempty" yaml:"name,omitempty"`
	BundleID     string        `json:"id" yaml:"id"`
	Timestamp    time.Time     `json:"timestamp,omitempty" yaml:"timestamp,omitempty"`
	Contributors []Contributor `json:"contributors" yaml:"contributors"`
	// contains filtered or unexported fields
}

LabelDescriptor describes a label

func NewLabelDescriptor 

func NewLabelDescriptor(opts ...LabelDescriptorOption) *LabelDescriptor

NewLabelDescriptor builds a new label descriptor

type LabelDescriptorOption 

type LabelDescriptorOption func(descriptor *LabelDescriptor)

LabelDescriptorOption is a functor to build label descriptors

func LabelContributor 

func LabelContributor(c Contributor) LabelDescriptorOption

LabelContributor sets a single contributor for the label

func LabelContributors 

func LabelContributors(c []Contributor) LabelDescriptorOption

LabelContributors sets a list of contributors for the label

func LabelName 

func LabelName(name string) LabelDescriptorOption

LabelName sets a name for the label

type LabelDescriptors 

type LabelDescriptors []LabelDescriptor

LabelDescriptors is a sortable slice of LabelDescriptor

func (LabelDescriptors) Last 

func (b LabelDescriptors) Last() LabelDescriptor

Last label in a LabelDescriptors slice

func (LabelDescriptors) Len 

func (b LabelDescriptors) Len() int

func (LabelDescriptors) Less 

func (b LabelDescriptors) Less(i, j int) bool

func (LabelDescriptors) Swap 

func (b LabelDescriptors) Swap(i, j int)

type RepoDescriptor 

type RepoDescriptor struct {
	Name        string      `json:"name,omitempty" yaml:"name,omitempty"`
	Description string      `json:"description,omitempty" yaml:"description,omitempty"`
	Timestamp   time.Time   `json:"timestamp,omitempty" yaml:"timestamp,omitempty"`
	Contributor Contributor `json:"contributor,omitempty" yaml:"contributor,omitempty"`
}

RepoDescriptor represents a commit which is a file tree with the changes to the repository.

type RepoDescriptors 

type RepoDescriptors []RepoDescriptor

RepoDescriptors is a sortable slice of RepoDescriptor

func (RepoDescriptors) Last 

func (b RepoDescriptors) Last() RepoDescriptor

Last returns the last entry in a slice of RepoDescriptors

func (RepoDescriptors) Len 

func (b RepoDescriptors) Len() int

func (RepoDescriptors) Less 

func (b RepoDescriptors) Less(i, j int) bool

func (RepoDescriptors) Swap 

func (b RepoDescriptors) Swap(i, j int)

type Run 

type Run struct {
	Name        string      `json:"name" yaml:"name"`               // Name for the run
	InstanceID  string      `json:"instanceid" yaml:"instanceid"`   // External ID used for a run. Not guaranteed to be unique.
	RunID       string      `json:"id" yaml:"id"`                   // Internally generated Unique id for an instance of a run. Each time the same Type+Name is run it results in a new ID.
	Timestamp   time.Time   `json:"timestamp" yaml:"timestamp"`     // Timestamp for when the run was created.
	Contributor Contributor `json:"contributor" yaml:"contributor"` // Contributor who created the run.
	Version     int         `json:"version" yaml:"version"`         // Version of the run Schema.
	Engine      RunEngine   `json:"engine" yaml:"engine"`           // Engine details for runs
}

Run is an unit of computation that is executed. A run consists of multiple stages where each stage has a set of container that read in data and generate data.

type RunEngine 

type RunEngine struct {
	RunFile string `json:"runfile" yaml:"runfile"` // Raw file describing the run. Example: A yaml file.
	Type    string `json:"type" yaml:"type"`       // The entity that executes the run.
}

RunEngine describes the details for the type of Run being executed and a runfile describing the run. Example: The yaml file generated by the pipelines tool for argo.

type RunStatus 

type RunStatus struct {
	Successful bool   `json:"successful" yaml:"successful"` // Successful run set to true if yes.
	Log        string `json:"log" yaml:"log"`               // Log for the final status, can be the entire status as reported by the pipeline engine.
}

RunStatus records the final status for a run.

type SplitDescriptor 

type SplitDescriptor struct {
	SplitID               string        `json:"splitID" yaml:"splitID"`
	StartTime             time.Time     `json:"startTime" yaml:"startTime"`                 // documentary: when the split was started
	EndTime               time.Time     `json:"endTime,omitempty" yaml:"endTime,omitempty"` // documentary: the split completion time (i.e. when the split has reached a terminal state)
	State                 SplitState    `json:"state" yaml:"state"`                         // the state of this split
	Contributors          []Contributor `json:"contributors" yaml:"contributors"`           // contributors to include in the resulting bundle
	GenerationID          string        `json:"generationID" yaml:"generationID"`           // unique location of index files used in final state (other possibly written locations are ignored)
	SplitEntriesFileCount uint64        `json:"count" yaml:"count"`                         // number of index files in this split
	Tag                   string        `json:"tag,omitempty" yaml:"tag,omitempty"`         // user-defined tag to taint logs
	// contains filtered or unexported fields
}

SplitDescriptor models the metadata about a given split within a diamond

func NewSplitDescriptor 

func NewSplitDescriptor(opts ...SplitDescriptorOption) *SplitDescriptor

NewSplitDescriptor builds a new SplitDescriptor

type SplitDescriptorOption 

type SplitDescriptorOption func(*SplitDescriptor)

SplitDescriptorOption defines an option to build a SplitDescriptor

func SplitClone 

func SplitClone(m SplitDescriptor) SplitDescriptorOption

SplitClone clones from a SplitDescriptor

func SplitContributor 

func SplitContributor(c Contributor) SplitDescriptorOption

SplitContributor defines a single contributor for a SplitDescriptor

func SplitContributors 

func SplitContributors(c []Contributor) SplitDescriptorOption

SplitContributors defines the list of contributors for a SplitDescriptor

func SplitID 

func SplitID(id string) SplitDescriptorOption

SplitID sets the splitID of a SplitDescriptor

func SplitTag 

func SplitTag(tag string) SplitDescriptorOption

SplitTag sets an informative tag on the split

type SplitDescriptors 

type SplitDescriptors []SplitDescriptor

SplitDescriptors is a sortable slice of SplitDescriptor

func (SplitDescriptors) Last 

func (b SplitDescriptors) Last() SplitDescriptor

Last bundle descriptor in slice

func (SplitDescriptors) Len 

func (b SplitDescriptors) Len() int

func (SplitDescriptors) Less 

func (b SplitDescriptors) Less(i, j int) bool

func (SplitDescriptors) Swap 

func (b SplitDescriptors) Swap(i, j int)

type SplitState 

type SplitState string

SplitState models the running status of an ungoing diamond split 

const (
	// SplitDone is the state of a completed split. This is a terminal state.
	SplitDone SplitState = done

	// SplitRunning is the state of running split
	SplitRunning SplitState = running
)

func (SplitState) IsValid 

func (s SplitState) IsValid() bool

IsValid checks the value of a diamond state

func (SplitState) String 

func (s SplitState) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.