upload

package
v0.0.0-...-58c64f7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 26, 2025 License: Apache-2.0 Imports: 39 Imported by: 0

Documentation

Overview

Package upload manages snapshot uploads.

Index

Constants

View Source 
const (
	IncompleteReasonCheckpoint   = "checkpoint"
	IncompleteReasonCanceled     = "canceled"
	IncompleteReasonLimitReached = "limit reached"
)

reasons why a snapshot is incomplete.

View Source 
const (
	// EstimationTypeClassic represents old way of estimation, which assumes iterating over all files.
	EstimationTypeClassic = "classic"
	// EstimationTypeRough represents new way of estimation, which looks into filesystem stats to get amount of data.
	EstimationTypeRough = "rough"
	// EstimationTypeAdaptive is a combination of new and old approaches. If the estimated file count is high,
	// it will use a rough estimation. If the count is low, it will switch to the classic method.
	EstimationTypeAdaptive = "adaptive"

	// AdaptiveEstimationThreshold is the point at which the classic estimation is used instead of the rough estimation.
	AdaptiveEstimationThreshold = 300000
)
View Source 
const DefaultCheckpointInterval = 45 * time.Minute

DefaultCheckpointInterval is the default frequency of mid-upload checkpointing.

Variables

This section is empty.

Functions

func Estimate 

func Estimate(ctx context.Context, entry fs.Directory, policyTree *policy.Tree, progress EstimateProgress, maxExamplesPerBucket int) error

Estimate walks the provided directory tree and invokes provided progress callback as it discovers items to be snapshotted.

Types

type Counters 

type Counters struct {
	// +checkatomic
	TotalCachedBytes int64 `json:"cachedBytes"`
	// +checkatomic
	TotalHashedBytes int64 `json:"hashedBytes"`
	// +checkatomic
	TotalUploadedBytes int64 `json:"uploadedBytes"`

	// +checkatomic
	EstimatedBytes int64 `json:"estimatedBytes"`

	// +checkatomic
	TotalCachedFiles int32 `json:"cachedFiles"`
	// +checkatomic
	TotalHashedFiles int32 `json:"hashedFiles"`

	// +checkatomic
	TotalExcludedFiles int32 `json:"excludedFiles"`
	// +checkatomic
	TotalExcludedDirs int32 `json:"excludedDirs"`

	// +checkatomic
	FatalErrorCount int32 `json:"errors"`
	// +checkatomic
	IgnoredErrorCount int32 `json:"ignoredErrors"`
	// +checkatomic
	EstimatedFiles int64 `json:"estimatedFiles"`

	CurrentDirectory string `json:"directory"`

	LastErrorPath string `json:"lastErrorPath"`
	LastError     string `json:"lastError"`
}

Counters represents a snapshot of upload counters.

type CountingUploadProgress 

type CountingUploadProgress struct {
	NullUploadProgress
	// contains filtered or unexported fields
}

CountingUploadProgress is an implementation of UploadProgress that accumulates counters.

func (*CountingUploadProgress) CachedFile 

func (p *CountingUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*CountingUploadProgress) Error 

func (p *CountingUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*CountingUploadProgress) EstimatedDataSize 

func (p *CountingUploadProgress) EstimatedDataSize(numFiles, numBytes int64)

EstimatedDataSize implements UploadProgress.

func (*CountingUploadProgress) ExcludedDir 

func (p *CountingUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*CountingUploadProgress) ExcludedFile 

func (p *CountingUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*CountingUploadProgress) FinishedFile 

func (p *CountingUploadProgress) FinishedFile(fname string, err error)

FinishedFile implements UploadProgress.

func (*CountingUploadProgress) FinishedHashingFile 

func (p *CountingUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*CountingUploadProgress) HashedBytes 

func (p *CountingUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*CountingUploadProgress) Snapshot 

func (p *CountingUploadProgress) Snapshot() Counters

Snapshot captures current snapshot of the upload.

func (*CountingUploadProgress) StartedDirectory 

func (p *CountingUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*CountingUploadProgress) UITaskCounters 

func (p *CountingUploadProgress) UITaskCounters(final bool) map[string]uitask.CounterValue

UITaskCounters returns UI task counters.

func (*CountingUploadProgress) UploadStarted 

func (p *CountingUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*CountingUploadProgress) UploadedBytes 

func (p *CountingUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type EstimateProgress 

type EstimateProgress interface {
	Processing(ctx context.Context, dirname string)
	Error(ctx context.Context, filename string, err error, isIgnored bool)
	Stats(ctx context.Context, s *snapshot.Stats, includedFiles, excludedFiles SampleBuckets, excludedDirs []string, final bool)
}

EstimateProgress must be provided by the caller of Estimate to report results.

type EstimationController 

type EstimationController interface {
	Cancel()
	Wait()
}

EstimationController defines an interface which has to be used to cancel or wait for running estimation.

type EstimationDoneFn 

type EstimationDoneFn func(int64, int64)

EstimationDoneFn represents the signature of the callback function which will be invoked when an estimation is done.

type EstimationParameters 

type EstimationParameters struct {
	Type              string
	AdaptiveThreshold int64
}

EstimationParameters represents parameters to be used for estimation.

type EstimationStarter 

type EstimationStarter interface {
	StartEstimation(ctx context.Context, cb EstimationDoneFn)
}

EstimationStarter defines an interface that is used to start an estimation of the size of data to be uploaded.

type Estimator 

type Estimator interface {
	EstimationStarter
	EstimationController
}

Estimator interface combines EstimationStarter and EstimationController interfaces. It represents the objects that can both initiate and control an estimation process.

func NewEstimator 

func NewEstimator(
	entry fs.Directory,
	policyTree *policy.Tree,
	estimationParams EstimationParameters,
	logger logging.Logger,
	options ...EstimatorOption,
) Estimator

NewEstimator returns instance of estimator.

type EstimatorOption 

type EstimatorOption func(Estimator)

EstimatorOption is an option which could be used to customize estimator behavior.

func WithVolumeSizeInfoFn 

func WithVolumeSizeInfoFn(fn VolumeSizeInfoFn) EstimatorOption

WithVolumeSizeInfoFn returns EstimatorOption which allows to pass custom GetVolumeSizeInfo implementation.

type NoOpEstimationController 

type NoOpEstimationController struct{}

NoOpEstimationController is a default implementation of the EstimationController interface. It's used in cases where no estimation operation is running and hence, its methods are no-ops.

func (*NoOpEstimationController) Cancel 

func (c *NoOpEstimationController) Cancel()

Cancel is a no-op function to satisfy the EstimationController interface.

func (*NoOpEstimationController) Wait 

func (c *NoOpEstimationController) Wait()

Wait is a no-op function to satisfy the EstimationController interface.

type NullUploadProgress 

type NullUploadProgress struct{}

NullUploadProgress is an implementation of UploadProgress that does not produce any output.

func (*NullUploadProgress) CachedFile 

func (p *NullUploadProgress) CachedFile(fname string, numBytes int64)

CachedFile implements UploadProgress.

func (*NullUploadProgress) Enabled 

func (p *NullUploadProgress) Enabled() bool

Enabled implements UploadProgress, always returns false.

func (*NullUploadProgress) Error 

func (p *NullUploadProgress) Error(path string, err error, isIgnored bool)

Error implements UploadProgress.

func (*NullUploadProgress) EstimatedDataSize 

func (p *NullUploadProgress) EstimatedDataSize(fileCount, totalBytes int64)

EstimatedDataSize implements UploadProgress.

func (*NullUploadProgress) EstimationParameters 

func (p *NullUploadProgress) EstimationParameters() EstimationParameters

EstimationParameters implements UploadProgress.

func (*NullUploadProgress) ExcludedDir 

func (p *NullUploadProgress) ExcludedDir(dirname string)

ExcludedDir implements UploadProgress.

func (*NullUploadProgress) ExcludedFile 

func (p *NullUploadProgress) ExcludedFile(fname string, numBytes int64)

ExcludedFile implements UploadProgress.

func (*NullUploadProgress) FinishedDirectory 

func (p *NullUploadProgress) FinishedDirectory(dirname string)

FinishedDirectory implements UploadProgress.

func (*NullUploadProgress) FinishedFile 

func (p *NullUploadProgress) FinishedFile(fname string, err error)

FinishedFile implements UploadProgress.

func (*NullUploadProgress) FinishedHashingFile 

func (p *NullUploadProgress) FinishedHashingFile(fname string, numBytes int64)

FinishedHashingFile implements UploadProgress.

func (*NullUploadProgress) HashedBytes 

func (p *NullUploadProgress) HashedBytes(numBytes int64)

HashedBytes implements UploadProgress.

func (*NullUploadProgress) HashingFile 

func (p *NullUploadProgress) HashingFile(fname string)

HashingFile implements UploadProgress.

func (*NullUploadProgress) StartedDirectory 

func (p *NullUploadProgress) StartedDirectory(dirname string)

StartedDirectory implements UploadProgress.

func (*NullUploadProgress) UploadFinished 

func (p *NullUploadProgress) UploadFinished()

UploadFinished implements UploadProgress.

func (*NullUploadProgress) UploadStarted 

func (p *NullUploadProgress) UploadStarted()

UploadStarted implements UploadProgress.

func (*NullUploadProgress) UploadedBytes 

func (p *NullUploadProgress) UploadedBytes(numBytes int64)

UploadedBytes implements UploadProgress.

type Progress 

type Progress interface {
	// Enabled returns true when progress is enabled, false otherwise.
	Enabled() bool

	// UploadStarted is emitted once at the start of an upload
	UploadStarted()

	// UploadFinished is emitted once at the end of an upload
	UploadFinished()

	// CachedFile is emitted whenever uploader reuses previously uploaded entry without hashing the file.
	CachedFile(path string, size int64)

	// HashingFile is emitted at the beginning of hashing of a given file.
	HashingFile(fname string)

	// ExcludedFile is emitted when a file is excluded.
	ExcludedFile(fname string, size int64)

	// ExcludedDir is emitted when a directory is excluded.
	ExcludedDir(dirname string)

	// FinishedHashingFile is emitted at the end of hashing of a given file.
	FinishedHashingFile(fname string, numBytes int64)

	// FinishedFile is emitted when the uploader is done with a file, regardless of if it was hashed
	// or cached. If an error was encountered it reports that too. A call to FinishedFile gives no
	// information about the reachability of the file in checkpoints that may occur close to the
	// time this function is called.
	FinishedFile(fname string, err error)

	// HashedBytes is emitted while hashing any blocks of bytes.
	HashedBytes(numBytes int64)

	// Error is emitted when an error is encountered.
	Error(path string, err error, isIgnored bool)

	// UploadedBytes is emitted whenever bytes are written to the blob storage.
	UploadedBytes(numBytes int64)

	// StartedDirectory is emitted whenever a directory starts being uploaded.
	StartedDirectory(dirname string)

	// FinishedDirectory is emitted whenever a directory is finished uploading.
	FinishedDirectory(dirname string)

	// EstimationParameters returns settings to be used for estimation
	EstimationParameters() EstimationParameters

	// EstimatedDataSize is emitted whenever the size of upload is estimated.
	EstimatedDataSize(fileCount int64, totalBytes int64)
}

Progress is invoked by uploader to report status of file and directory uploads.

type SampleBucket 

type SampleBucket struct {
	MinSize   int64    `json:"minSize"`
	Count     int      `json:"count"`
	TotalSize int64    `json:"totalSize"`
	Examples  []string `json:"examples,omitempty"`
}

SampleBucket keeps track of count and total size of files above in certain size range and includes small number of examples of such files.

type SampleBuckets 

type SampleBuckets []*SampleBucket

SampleBuckets is a collection of buckets for interesting file sizes sorted in descending order.

type Uploader 

type Uploader struct {
	Progress Progress

	// automatically cancel the Upload after certain number of bytes
	MaxUploadBytes int64

	// probability with cached entries will be ignored, must be [0..100]
	// 0=always use cached object entries if possible
	// 100=never use cached entries
	ForceHashPercentage float64

	// Number of files to hash and upload in parallel.
	ParallelUploads int

	// Enable snapshot actions
	EnableActions bool

	// override the directory log level and entry log verbosity.
	OverrideDirLogDetail   *policy.LogDetail
	OverrideEntryLogDetail *policy.LogDetail

	// Fail the entire snapshot on source file/directory error.
	FailFast bool

	// How frequently to create checkpoint snapshot entries.
	CheckpointInterval time.Duration

	// When set to true, do not ignore any files, regardless of policy settings.
	DisableIgnoreRules bool

	// Labels to apply to every checkpoint made for this snapshot.
	CheckpointLabels map[string]string
	// contains filtered or unexported fields
}

Uploader supports efficient uploading files and directories to repository.

func NewUploader 

func NewUploader(r repo.RepositoryWriter) *Uploader

NewUploader creates new Uploader object for a given repository.

func (*Uploader) Cancel 

func (u *Uploader) Cancel()

Cancel requests cancellation of an upload that's in progress. Will typically result in an incomplete snapshot.

func (*Uploader) IsCanceled 

func (u *Uploader) IsCanceled() bool

IsCanceled returns true if the upload is canceled.

func (*Uploader) Upload 

func (u *Uploader) Upload(
	ctx context.Context,
	source fs.Entry,
	policyTree *policy.Tree,
	sourceInfo snapshot.SourceInfo,
	previousManifests ...*snapshot.Manifest,
) (*snapshot.Manifest, error)

Upload uploads contents of the specified filesystem entry (file or directory) to the repository and returns snapshot.Manifest with statistics. Old snapshot manifest, when provided can be used to speed up uploads by utilizing hash cache.

type VolumeSizeInfoFn 

type VolumeSizeInfoFn func(string) (vsi.VolumeSizeInfo, error)

VolumeSizeInfoFn represents a function type which is used to retrieve volume size information.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.