artifact

package
Version: v2.34.0-RC1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 26, 2021 License: Apache-2.0, BSD-3-Clause, MIT Imports: 22 Imported by: 0

Documentation

Overview

Package artifact contains utilities for staging and retrieving artifacts.

Index

Constants

View Source
const (
	URNFileArtifact        = "beam:artifact:type:file:v1"
	URNPipRequirementsFile = "beam:artifact:role:pip_requirements_file:v1"
	URNStagingTo           = "beam:artifact:role:staging_to:v1"
	NoArtifactsStaged      = "__no_artifacts_staged__"
)

TODO(lostluck): 2018/05/28 Extract these from their enum descriptors in the pipeline_v1 proto

Variables

This section is empty.

Functions

func Commit

Commit commits a manifest with the given staged artifacts. It returns the staging token, if successful.

func LegacyMultiRetrieve

func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error

LegacyMultiRetrieve is exported for testing.

func Materialize

func Materialize(ctx context.Context, endpoint string, dependencies []*pipepb.ArtifactInformation, rt string, dest string) ([]*pipepb.ArtifactInformation, error)

Materialize is a convenience helper for ensuring that all artifacts are present and uncorrupted. It interprets each artifact name as a relative path under the dest directory. It does not retrieve valid artifacts already present. TODO(BEAM-9577): Return a mapping of filename to dependency, rather than []*jobpb.ArtifactMetadata. TODO(BEAM-9577): Leverage richness of roles rather than magic names to understand artifacts.

func MultiRetrieve

func MultiRetrieve(ctx context.Context, cpus int, list []retrievable, dest string) error

MultiRetrieve retrieves multiple artifacts concurrently, using at most 'cpus' goroutines. It retries each artifact a few times. Convenience wrapper.

func MultiStage

MultiStage stages a set of local files with the given keys. It returns the full artifact metadate. It retries each artifact a few times. Convenience wrapper.

func MustExtractFilePayload

func MustExtractFilePayload(artifact *pipepb.ArtifactInformation) (string, string)

func Retrieve

Retrieve checks whether the given artifact is already successfully retrieved. If not, it retrieves into the dest directory. It overwrites any previous retrieval attempt and may leave a corrupt/partial local file on failure.

func Stage

Stage stages a local file as an artifact with the given key. It computes the SHA256 and returns the full artifact metadata.

func StageDir

StageDir stages a local directory with relative path keys. Convenience wrapper.

Types

type KeyedFile

type KeyedFile struct {
	Key, Filename string
}

KeyedFile is a key and filename pair.

Directories

Path Synopsis
Package gcsproxy contains artifact staging and retrieval servers backed by GCS.
Package gcsproxy contains artifact staging and retrieval servers backed by GCS.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL