baur

package
v2.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2022 License: GPL-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package baur implements an incremental task runner.

A directory and child-directories containing baur configuration files is called a repository. The root of the repository contains the RepositoryCfgFile. The root and each child directory can contain an application with an AppCfgFile.

Each AppCfgFile can define one or more tasks. A task consist of a Command to run, Input definitions and optionally Output definitions. An Input is a file or string that when it changes, it changes the result of the executed command. An output is a something that is created when the task is executed. Outputs define remote upload locations to which they should be uploaded too. The results of task runs together with the state of the inputs are recorded in a storage. The state of inputs is tracked by calculating a digest for all inputs of a task. With the records about past task runs in the storage and the digest of the inputs of a task, baur evaluates if a task has been run with the same inputs in the past already. If it has not, it's executions is pending.

Basic Workflow

- Loader: Locate and load a repository configuration file, discover applications and load and parse their configuration files.

- TaskStatusEvaluator: Query the storage and determine which applications have not been run before with their current inputs.

- TaskRunner: Execute the commands for the tasks that should be run.

- Uploader: Uploads the outputs that the tasks produced.

- StoreRun: Record the task executions the uploaded outputs in the database.

Index

Constants

View Source 
const AppCfgFile = ".app.toml"

AppCfgFile is the name of application configuration files.

View Source 
const RepositoryCfgFile = ".baur.toml"

RepositoryCfgFile is the name of the repository configuration file.

Variables

View Source 
var ErrTaskRunSkipped = errors.New("task run skipped")

ErrTaskRunSkipped is returned when a task run was skipped instead of executed.

Functions

func FindRepositoryCfg 

func FindRepositoryCfg(dir string) (string, error)

FindRepositoryCfg searches for a repository config file. The search starts in dir and traverses the parent directory down to the root. It returns the path to the first found repository configuration file.

func SortAppsByName 

func SortAppsByName(apps []*App)

SortAppsByName sorts the slice by application names.

func SortTasksByID 

func SortTasksByID(tasks []*Task)

SortTasksByID sorts the tasks slice by task IDs.

func StoreRun 

func StoreRun(
	ctx context.Context,
	storer storage.Storer,
	vcsState vcs.StateFetcher,
	task *Task,
	inputs *Inputs,
	runResult *RunResult,
	uploads []*UploadResult,
) (int, error)

StoreRun stores the result of a task run in a baur storage.

Types

type App 

type App struct {
	RelPath string
	Path    string
	Name    string
	// contains filtered or unexported fields
}

App represents an Application.

func NewApp 

func NewApp(appCfg *cfg.App, repositoryRootPath string) (*App, error)

NewApp instantiates an App object based on an app configuration.

func (*App) String 

func (a *App) String() string

String returns the name of the app.

func (*App) Tasks 

func (a *App) Tasks() []*Task

Tasks instantiates Task objects for each defined task in the app's configuration.

type CfgUpgrader 

type CfgUpgrader struct {
	// contains filtered or unexported fields
}

CfgUpgrader converts baur configurations files from a previous format to the current one.

func NewCfgUpgrader 

func NewCfgUpgrader(repositoryRootDir string) *CfgUpgrader

NewCfgUpgrader returns an new CfgUpgrader to upgrade configuration files in repositoryRootDir.

func (*CfgUpgrader) Upgrade 

func (u *CfgUpgrader) Upgrade() error

Upgrade upgrades all baur configuration files. Of all changed files a backup copy is created with the same filename and a ".bak" suffix. Each upgraded configuration file is validated by running the responsible validate() method from the cfg package.

type DiffType 

type DiffType int

DiffType represents the difference betweeen two baur Inputs. 

const (
	DigestMismatch DiffType = iota
	Removed
	Added
)

func (DiffType) String 

func (d DiffType) String() string

type DockerImgUploader 

type DockerImgUploader interface {
	Upload(image, registryAddr, repository, tag string) (string, error)
}

DockerImgUploader is an interface for uploading docker images to a docker registry.

type DockerInfoClient 

type DockerInfoClient interface {
	// Size returns the size of an image in bytes
	SizeBytes(imageID string) (int64, error)
	// Exists returns true and no error if an image with the image ID imageID exists locally.
	Exists(imageID string) (bool, error)
}

DockerInfoClient is an interface fo retrieving information about a docker image.

type FileCopyUploader 

type FileCopyUploader interface {
	Upload(src string, dst string) (string, error)
}

FileCopyUploader is an interface for copying files from one directory to another.

type Input 

type Input interface {
	Digest() (*digest.Digest, error)
	String() string
}

Input represents an input

func AsInputStrings 

func AsInputStrings(strs ...string) []Input

AsInputStrings returns InputStrings for all elements in strs.

type InputDiff 

type InputDiff struct {
	State   DiffType
	Path    string
	Digest1 string
	Digest2 string
}

InputDiff represents the result of a DiffInputs operation.

func DiffInputs 

func DiffInputs(a, b *Inputs) ([]*InputDiff, error)

DiffInputs returns the differences between two sets of Inputs. The Input.String() is used as the key to identify each Input.

type InputFile 

type InputFile struct {
	// contains filtered or unexported fields
}

InputFile represent a file.

func NewInputFile 

func NewInputFile(repoRootPath, relPath string) *InputFile

NewInputFile returns a new input file

func (*InputFile) AbsPath 

func (f *InputFile) AbsPath() string

func (*InputFile) CalcDigest 

func (f *InputFile) CalcDigest() (*digest.Digest, error)

CalcDigest calculates the digest of the file. The Digest is the sha384 sum of the repoRelPath and the content of the file.

func (*InputFile) Digest 

func (f *InputFile) Digest() (*digest.Digest, error)

Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called.

func (*InputFile) RelPath 

func (f *InputFile) RelPath() string

RelPath returns the path of the file relative to the baur repository root.

func (*InputFile) String 

func (f *InputFile) String() string

String returns RelPath()

type InputResolver 

type InputResolver struct {
	// contains filtered or unexported fields
}

InputResolver resolves input definitions of a task to concrete files.

func NewCachingInputResolver 

func NewCachingInputResolver() *InputResolver

NewCachingInputResolver returns an InputResolver that caches resolver results.

func (*InputResolver) Resolve 

func (i *InputResolver) Resolve(ctx context.Context, repositoryDir string, task *Task) ([]Input, error)

Resolve resolves the input definition of the task to concrete Files. If an input definition does not resolve to >=1 paths, an error is returned. The resolved Files are deduplicated.

type InputString 

type InputString struct {
	// contains filtered or unexported fields
}

InputString represents a string

func NewInputString 

func NewInputString(val string) *InputString

NewInputString returns a new InputString

func (*InputString) Digest 

func (i *InputString) Digest() (*digest.Digest, error)

Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called and it's return values are returned.

func (*InputString) String 

func (i *InputString) String() string

String returns it's string representation (string:VAL)

func (*InputString) Value 

func (i *InputString) Value() string

Value returns the string that the input represents.

type Inputs 

type Inputs struct {
	// contains filtered or unexported fields
}

Inputs are resolved Inputs of a task.

func NewInputs 

func NewInputs(in []Input) *Inputs

NewInputs returns an new Inputs

func (*Inputs) Add 

func (in *Inputs) Add(inputs []Input) *Inputs

Add adds elements in inputs to in and returns *in

func (*Inputs) Digest 

func (in *Inputs) Digest() (*digest.Digest, error)

Digest returns a summarized digest over all Inputs. On the first call the digest is calculated, on subsequent calls the stored digest is returned.

func (*Inputs) Inputs 

func (in *Inputs) Inputs() []Input

Inputs returns all stored Inputs

type Loader 

type Loader struct {
	// contains filtered or unexported fields
}

Loader discovers and instantiates apps and tasks.

func NewLoader 

func NewLoader(repoCfg *cfg.Repository, gitCommitIDFunc func() (string, error), logger Logger) (*Loader, error)

NewLoader instantiates a Loader. When an app config is loaded the DefaultResolvers are applied on the content before they are merged with their includes. The gitCommitIDFunc is used as config resolved to resolve $GITCOMMIT variables.

func (*Loader) LoadApps 

func (a *Loader) LoadApps(specifier ...string) ([]*App, error)

LoadApps loads the apps that match the passed specifiers. Valid specifiers are: - application directory path - <APP-NAME> - '*' If no specifier is passed all apps are returned. If multiple specifiers match the same app, it's only returned 1x in the returned slice.

func (*Loader) LoadTasks 

func (a *Loader) LoadTasks(specifier ...string) ([]*Task, error)

LoadTasks loads the tasks of apps that match the passed specifier. Specifier format is (<APP-SPEC>[.<TASK-SPEC>])|PATH <APP-SPEC> is:

  • <APP-NAME> or
  • '*'

<TASK-SPEC> is:

  • Task Name or
  • '*'

If no specifier is passed all tasks of all apps are returned. If multiple specifiers match the same task, it's only returned 1x in the returned slice.

type Logger 

type Logger interface {
	Debugf(format string, v ...interface{})
}

type Output 

type Output interface {
	Name() string
	String() string
	Exists() (bool, error)
	SizeBytes() (uint64, error)
	Digest() (*digest.Digest, error)
	Type() OutputType
}

Output is an interface for an output that is produced by a task run.

func OutputsFromTask 

func OutputsFromTask(dockerClient DockerInfoClient, task *Task) ([]Output, error)

OutputsFromTask returns the Outputs that running the task produces. If the outputs do not exist, the function might fail.

type OutputDockerImage 

type OutputDockerImage struct {
	ImageID            string
	UploadDestinations []*UploadInfoDocker
	// contains filtered or unexported fields
}

OutputDockerImage is a docker image artifact.

func NewOutputDockerImageFromIIDFile 

func NewOutputDockerImageFromIIDFile(
	dockerClient DockerInfoClient,
	name,
	iidfile string,
	uploadDest []*UploadInfoDocker,
) (*OutputDockerImage, error)

NewOutputDockerImageFromIIDFile instantiates a new OutputDockerImage. name only acts as an identifier for the image. iidfile is the path to a file containing the ID of the docker image. uploadDest describes where the image should be uploaded to.

func (*OutputDockerImage) Digest 

func (d *OutputDockerImage) Digest() (*digest.Digest, error)

Digest returns the imageID as a digest object. The method always returns a nil error.

func (*OutputDockerImage) Exists 

func (d *OutputDockerImage) Exists() (bool, error)

func (*OutputDockerImage) Name 

func (d *OutputDockerImage) Name() string

func (*OutputDockerImage) SizeBytes 

func (d *OutputDockerImage) SizeBytes() (uint64, error)

func (*OutputDockerImage) String 

func (d *OutputDockerImage) String() string

func (*OutputDockerImage) Type 

func (d *OutputDockerImage) Type() OutputType

type OutputFile 

type OutputFile struct {
	UploadsS3       []*UploadInfoS3
	UploadsFilecopy []*UploadInfoFileCopy
	// contains filtered or unexported fields
}

OutputFile is a file created by a task run.

func NewOutputFile 

func NewOutputFile(name, absPath string, s3uploads []*UploadInfoS3, filecopyUploads []*UploadInfoFileCopy) *OutputFile

func (*OutputFile) AbsPath 

func (f *OutputFile) AbsPath() string

func (*OutputFile) CalcDigest 

func (f *OutputFile) CalcDigest() (*digest.Digest, error)

CalcDigest calculates the digest of the file. The Digest is the sha384 sum of the content of the file.

func (*OutputFile) Digest 

func (f *OutputFile) Digest() (*digest.Digest, error)

Digest returns the previous calculated digest. If the digest wasn't calculated yet, CalcDigest() is called.

func (*OutputFile) Exists 

func (f *OutputFile) Exists() (bool, error)

func (*OutputFile) Name 

func (f *OutputFile) Name() string

func (*OutputFile) SizeBytes 

func (f *OutputFile) SizeBytes() (uint64, error)

func (*OutputFile) String 

func (f *OutputFile) String() string

func (*OutputFile) Type 

func (f *OutputFile) Type() OutputType

type OutputType 

type OutputType int
const (
	DockerOutput OutputType = iota
	FileOutput
)

func (OutputType) String 

func (o OutputType) String() string

type Repository 

type Repository struct {
	Path        string
	CfgPath     string
	Cfg         *cfg.Repository
	SearchDepth int
}

Repository represents a baur repository.

func NewRepository 

func NewRepository(cfgPath string) (*Repository, error)

NewRepository parses the repository configuration file cfgPath and returns a Repository.

type RunResult 

type RunResult struct {
	*exec.Result
	StartTime time.Time
	StopTime  time.Time
}

RunResult represents the results of a task run.

type S3Uploader 

type S3Uploader interface {
	Upload(filepath, bucket, key string) (string, error)
}

S3Uploader is an interface for uploading files to AWS S3 buckets.

type Task 

type Task struct {
	RepositoryRoot string
	Directory      string

	AppName string

	Name             string
	Command          []string
	UnresolvedInputs *cfg.Input
	Outputs          *cfg.Output
	CfgFilepaths     []string
}

Task is a an execution step belonging to an app. A task has a set of Inputs that produce a set of outputs by executing it's Command.

func NewTask 

func NewTask(cfg *cfg.Task, appName, repositoryRootdir, workingDir string) *Task

NewTask returns a new Task.

func (*Task) HasInputs 

func (t *Task) HasInputs() bool

HasInputs returns true if Inputs are defined for the task

func (*Task) HasOutputs 

func (t *Task) HasOutputs() bool

HasOutputs returns true if outputs are defined for the task

func (*Task) ID 

func (t *Task) ID() string

ID returns APP-NAME.TASK-NAME.

func (*Task) String 

func (t *Task) String() string

String returns ID()

type TaskRunner 

type TaskRunner struct {
	// contains filtered or unexported fields
}

TaskRunner executes the command of a task.

func NewTaskRunner 

func NewTaskRunner() *TaskRunner

func (*TaskRunner) Run 

func (t *TaskRunner) Run(task *Task) (*RunResult, error)

Run executes the command of a task and returns the execution result. The output of the commands are logged with debug log level.

func (*TaskRunner) SkipRuns added in v2.1.0 

func (t *TaskRunner) SkipRuns(enabled bool)

SkipRuns can be enabled to skip all further executions of tasks by Run().

func (*TaskRunner) SkipRunsIsEnabled added in v2.1.0 

func (t *TaskRunner) SkipRunsIsEnabled() bool

SkipRunsIsEnabled returns true if SkipRuns is enabled.

type TaskStatus 

type TaskStatus int

TaskStatus describes the status of the task 

const (
	TaskStatusUndefined TaskStatus
	TaskStatusRunExist
	TaskStatusExecutionPending
)

func (TaskStatus) String 

func (b TaskStatus) String() string

type TaskStatusEvaluator 

type TaskStatusEvaluator struct {
	// contains filtered or unexported fields
}

TaskStatusEvaluator determines if a task already run with the same set of inputs in the past.

func NewTaskStatusEvaluator 

func NewTaskStatusEvaluator(
	repositoryDir string,
	store storage.Storer,
	inputResolver *InputResolver,
	inputStr []string,
	lookupInputStr string,
) *TaskStatusEvaluator

NewTaskStatusEvaluator returns a new TaskSNewTaskStatusEvaluator.

func (*TaskStatusEvaluator) Status 

func (t *TaskStatusEvaluator) Status(ctx context.Context, task *Task) (TaskStatus, *Inputs, *storage.TaskRunWithID, error)

Status resolves the inputs of the task, calculates the total input digest and checks in the storage if a run record for the task and total input digest already exist. If TaskStatusExecutionPending is returned, the returned TaskRunWithID is nil.

type UploadInfo 

type UploadInfo interface {
	String() string
}

type UploadInfoDocker 

type UploadInfoDocker struct {
	*cfg.DockerImageRegistryUpload
}

func (*UploadInfoDocker) String 

func (d *UploadInfoDocker) String() string

type UploadInfoFileCopy 

type UploadInfoFileCopy struct {
	*cfg.FileCopy
}

func (*UploadInfoFileCopy) String 

func (f *UploadInfoFileCopy) String() string

type UploadInfoS3 

type UploadInfoS3 struct {
	*cfg.S3Upload
}

func (*UploadInfoS3) String 

func (s *UploadInfoS3) String() string

type UploadMethod 

type UploadMethod int
const (
	UploadMethodS3 UploadMethod = iota
	UploadMethodFilecopy
	UploadMethodDocker
)

type UploadResult 

type UploadResult struct {
	// Output is the output that was uploaded.
	Output Output
	URL    string
	Start  time.Time
	Stop   time.Time
	Method UploadMethod
}

UploadResult is the result of an upload operation.

type UploadResultFn 

type UploadResultFn func(Output, *UploadResult)

UploadResultFn is a function that is called after an upload finishes.

type UploadStartFn 

type UploadStartFn func(Output, UploadInfo)

UploadStartFn is a function that is called before the upload operation starts.

type Uploader 

type Uploader struct {
	// contains filtered or unexported fields
}

Uploader uploads outputs, produced by task run, to remote locations.

func NewUploader 

func NewUploader(dockerClient DockerImgUploader, s3client S3Uploader, filecopyUploader FileCopyUploader) *Uploader

func (*Uploader) Upload 

func (u *Uploader) Upload(output Output, uploadStartCb UploadStartFn, resultCb UploadResultFn) error

Upload uploads an output to remote locations. Output must be a *OutputDockerImage or *OutputFile type storing or more upload locations. Immediately before the upload starts uploadStartCb is called, when the upload finished resultCb is called.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.