Documentation

Overview

    Package artifact contains utilities for staging and retrieving artifacts.

    Index

    Constants

    View Source
    const (
    	URNStagingTo      = "beam:artifact:role:staging_to:v1"
    	NoArtifactsStaged = "__no_artifacts_staged__"
    )

      TODO(lostluck): 2018/05/28 Extract these from their enum descriptors in the pipeline_v1 proto

      Variables

      This section is empty.

      Functions

      func Commit

        Commit commits a manifest with the given staged artifacts. It returns the staging token, if successful.

        func LegacyMultiRetrieve

        func LegacyMultiRetrieve(ctx context.Context, client jobpb.LegacyArtifactRetrievalServiceClient, cpus int, list []*jobpb.ArtifactMetadata, rt string, dest string) error

          LegacyMultiRetrieve is exported for testing.

          func Materialize

          func Materialize(ctx context.Context, endpoint string, dependencies []*pipepb.ArtifactInformation, rt string, dest string) ([]*jobpb.ArtifactMetadata, error)

            Materialize is a convenience helper for ensuring that all artifacts are present and uncorrupted. It interprets each artifact name as a relative path under the dest directory. It does not retrieve valid artifacts already present. TODO(BEAM-9577): Return a mapping of filename to dependency, rather than []*jobpb.ArtifactMetadata. TODO(BEAM-9577): Leverage richness of roles rather than magic names to understand artifacts.

            func MultiRetrieve

            func MultiRetrieve(ctx context.Context, cpus int, list []retrievable, dest string) error

              MultiRetrieve retrieves multiple artifacts concurrently, using at most 'cpus' goroutines. It retries each artifact a few times. Convenience wrapper.

              func MultiStage

                MultiStage stages a set of local files with the given keys. It returns the full artifact metadate. It retries each artifact a few times. Convenience wrapper.

                func Retrieve

                  Retrieve checks whether the given artifact is already successfully retrieved. If not, it retrieves into the dest directory. It overwrites any previous retrieval attempt and may leave a corrupt/partial local file on failure.

                  func Stage

                    Stage stages a local file as an artifact with the given key. It computes the SHA256 and returns the full artifact metadata.

                    func StageDir

                      StageDir stages a local directory with relative path keys. Convenience wrapper.

                      Types

                      type KeyedFile

                      type KeyedFile struct {
                      	Key, Filename string
                      }

                        KeyedFile is a key and filename pair.

                        Directories

                        Path Synopsis
                        Package gcsproxy contains artifact staging and retrieval servers backed by GCS.
                        Package gcsproxy contains artifact staging and retrieval servers backed by GCS.