gcs

package
v0.0.0-...-1738bad Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 10, 2024 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package gcs handles uploading files and raw data to blob storage and determines where in the GCS bucket data should go given a specific job specification

Index

Constants

View Source
const (
	// PRLogs is the name of the directory we put PR logs in.
	PRLogs = "pr-logs"
	// NonPRLogs is the name of directory we put logs that are not PR logs in.
	NonPRLogs = "logs"
)

Variables

This section is empty.

Functions

func AliasForSpec

func AliasForSpec(spec *downwardapi.JobSpec) string

AliasForSpec determines the GCS path aliases for a job spec

func LatestBuildForSpec

func LatestBuildForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) []string

LatestBuildForSpec determines the GCS path for storing the latest build id for a job. pathSegment can be nil so callers of this helper are not required to choose a path strategy but can still get back a result.

func LocalExport

func LocalExport(ctx context.Context, exportDir string, uploadTargets map[string]UploadFunc) error

LocalExport copies all of the data in the uploadTargets map to local files in parallel. The map is keyed on file path under the exportDir.

func PathForSpec

func PathForSpec(spec *downwardapi.JobSpec, pathSegment RepoPathBuilder) string

PathForSpec determines the GCS path prefix for files uploaded for a specific job spec

func RootForSpec

func RootForSpec(spec *downwardapi.JobSpec) string

RootForSpec determines the root GCS path for storing artifacts about the provided job.

func Upload

func Upload(ctx context.Context, bucket, gcsCredentialsFile, s3CredentialsFile string, compressFileTypes []string, uploadTargets map[string]UploadFunc) error

Upload uploads all the data in the uploadTargets map to blob storage in parallel. The map is keyed on blob storage path under the bucket. Files with an extension in the compressFileTypes list will be compressed prior to uploading

func WriterOptionsFromFileName

func WriterOptionsFromFileName(filename string) (string, io.WriterOptions)

WriterOptionsFromFileName guesses file attributes from the filename and returns the writerOptions and a simplified filename. For example, build-log.txt.gz would be:

Content-Type: text/plain; charset=utf-8
Content-Encoding: gzip

and the simplified filename would be build-log.txt (excluding the content encoding extension).

Types

type ReaderFunc

type ReaderFunc func() (io.ReadCloser, error)

type RepoPathBuilder

type RepoPathBuilder func(org, repo string) string

RepoPathBuilder builds GCS path segments and embeds defaulting behavior

func NewExplicitRepoPathBuilder

func NewExplicitRepoPathBuilder() RepoPathBuilder

NewExplicitRepoPathBuilder returns a builder that handles the path encoding where a path will always have an explicit "org_repo" path segment

func NewLegacyRepoPathBuilder

func NewLegacyRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder

NewLegacyRepoPathBuilder returns a builder that handles the legacy path encoding where a path will only contain an org or repo if they are non-default

func NewSingleDefaultRepoPathBuilder

func NewSingleDefaultRepoPathBuilder(defaultOrg, defaultRepo string) RepoPathBuilder

NewSingleDefaultRepoPathBuilder returns a builder that handles the legacy path encoding where a path will contain org and repo for all but one default repo

type UploadFunc

type UploadFunc func(writer dataWriter) error

UploadFunc knows how to upload into an object

func DataUpload

func DataUpload(newReader ReaderFunc) UploadFunc

DataUpload returns an UploadFunc which copies all data from src reader into GCS.

func DataUploadWithMetadata

func DataUploadWithMetadata(newReader ReaderFunc, metadata map[string]string) UploadFunc

DataUploadWithMetadata returns an UploadFunc which copies all data from src reader into GCS and also sets the provided metadata fields onto the object.

func DataUploadWithOptions

func DataUploadWithOptions(newReader ReaderFunc, attrs pkgio.WriterOptions) UploadFunc

DataUploadWithOptions returns an UploadFunc which copies all data from src reader into GCS and also sets the provided attributes on the object.

func FileUpload

func FileUpload(file string) UploadFunc

FileUpload returns an UploadFunc which copies all data from the file on disk to the GCS object

func FileUploadWithOptions

func FileUploadWithOptions(file string, opts pkgio.WriterOptions) UploadFunc

FileUploadWithOptions returns an UploadFunc which copies all data from the file on disk into GCS object and also sets the provided attributes on the object.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL