transfer

Documentation

Index

• Constants
• func ParseTransferSegment(def Definition) (TransferSegment, TransferSegment)
• type Channels
• type Committer
• type Definition
  • func ParseTransferDefinition(str string) (Definition, error)
• type DupeCheckLevel
  • func ParseDupeCheckLevel(str string) (res DupeCheckLevel, err error)
  • func (d *DupeCheckLevel) ToString() string
• type Transfer
  • func NewTransfer(source pipeline.SourcePipeline, target pipeline.TargetPipeline, readers int, ...) *Transfer
  • func (t *Transfer) SetTransferTracker(tracker *internal.TransferTracker)
  • func (t *Transfer) StartTransfer(dupeLevel DupeCheckLevel)
  • func (t *Transfer) WaitForCompletion() time.Time
• type TransferSegment
• type WaitGroups
• type Worker

Constants

const (
	FileToBlock Definition = "file-blockblob"
	HTTPToBlock            = "http-blockblob"
	BlobToFile             = "blob-file"
	FileToPage             = "file-pageblob"
	HTTPToPage             = "http-pageblob"
	HTTPToFile             = "http-file"
	BlobToBlock            = "blob-blockblob"
	BlobToPage             = "blob-pageblob"
	S3ToBlock              = "s3-blockblob"
	S3ToPage               = "s3-pageblob"
	PerfToBlock            = "perf-blockblob"
	PerfToPage             = "perf-pageblob"
	BlobToPerf             = "blob-perf"
)

The different transfers types

const (
	File      TransferSegment = "file"
	HTTP                      = "http"
	BlockBlob                 = "blockblob"
	PageBlob                  = "pageblob"
	S3                        = "s3"
	Perf                      = "perf"
	Blob                      = "blob"
	NA                        = "na"
)

valid transfer segments

const (
	//CommitEvent TODO
	CommitEvent internal.EventName = "commit"
	//BufferEvent TODO
	BufferEvent = "buffer"
	//DataWrittenEvent TODO
	DataWrittenEvent = "data-written"
	//WrittenPartEvent TODO
	WrittenPartEvent = "written-part"
)

const DupeCheckLevelStr = "None, ZeroOnly, Full" //TODO: package this better so it can stay in sync w/declaration

DupeCheckLevelStr list of duplicate blob check strategies

const (
	//WriteDurationEvent TODO
	WriteDurationEvent internal.EventName = "write-duration"
)

Functions

func ParseTransferSegment

func ParseTransferSegment(def Definition) (TransferSegment, TransferSegment)

ParseTransferSegment TODO

Types

type Channels

type Channels struct {
	ReadParts  chan pipeline.Part
	Parts      chan pipeline.Part
	Partitions chan pipeline.PartsPartition
}

Channels represents all the control channels in the transfer.

type Committer

type Committer struct {
	// contains filtered or unexported fields
}

type Definition

type Definition string

Definition represents the supported source and target combinations

func ParseTransferDefinition

func ParseTransferDefinition(str string) (Definition, error)

ParseTransferDefinition parses a Definition from a string.

type DupeCheckLevel

type DupeCheckLevel int

DupeCheckLevel -- degree to which we'll try to check for duplicate blocks

const (
	None     DupeCheckLevel = iota // Don't bother to check
	ZeroOnly                       // Only check for blocks with all zero bytes
	Full                           // Compute MD5s and look for matches on that
)

The different levels for duplicate blocks

func ParseDupeCheckLevel

func ParseDupeCheckLevel(str string) (res DupeCheckLevel, err error)

ParseDupeCheckLevel converts string to DupeCheckLevel

func (*DupeCheckLevel) ToString

func (d *DupeCheckLevel) ToString() string

ToString printable representation of duplicate level values. For now, handle error as fatal, shouldn't be possible

type Transfer

type Transfer struct {
	SourcePipeline   pipeline.SourcePipeline
	TargetPipeline   pipeline.TargetPipeline
	NumOfReaders     int
	NumOfWorkers     int
	TotalNumOfBlocks int
	TotalSize        uint64
	ThreadTarget     int
	SyncWaitGroups   *WaitGroups
	ControlChannels  *Channels
	// contains filtered or unexported fields
}

Transfer top data structure holding the state of the transfer.

func NewTransfer

func NewTransfer(source pipeline.SourcePipeline, target pipeline.TargetPipeline, readers int, workers int, blockSize uint64) *Transfer

NewTransfer creates a new Transfer, this will adjust the thread target and initialize the channels and the wait groups for the writers, readers and the committers

func (*Transfer) SetTransferTracker

func (t *Transfer) SetTransferTracker(tracker *internal.TransferTracker)

SetTransferTracker TODO

func (*Transfer) StartTransfer

func (t *Transfer) StartTransfer(dupeLevel DupeCheckLevel)

StartTransfer starts the readers and readers. Process and commits the results of the write operations. The duration timer is started.

func (*Transfer) WaitForCompletion

func (t *Transfer) WaitForCompletion() time.Time

WaitForCompletion blocks until the readers, writers and commit operations complete. Duration property is set and returned.

type TransferSegment

type TransferSegment string

TransferSegment source and target types

type WaitGroups

type WaitGroups struct {
	Readers sync.WaitGroup
	Workers sync.WaitGroup
	Commits sync.WaitGroup
}

WaitGroups holds all wait groups involved in the transfer.

type Worker

type Worker struct {
	// contains filtered or unexported fields
}

Worker represents a worker routine that transfers data to a target and commits the list if applicable.