targets

Documentation

Index

• Constants
• func NewAzureBlock(accountName string, accountKey string, container string) pipeline.TargetPipeline
• func NewAzurePage(accountName string, accountKey string, container string) pipeline.TargetPipeline
• func NewMultiFile(overwrite bool, numberOfHandles int) pipeline.TargetPipeline
• type AzureBlock
  • func (t *AzureBlock) CommitList(listInfo *pipeline.TargetCommittedListInfo, numberOfBlocks int, ...) (msg string, err error)
  • func (t *AzureBlock) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)
  • func (t *AzureBlock) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)
  • func (t *AzureBlock) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)
• type AzurePage
  • func (t *AzurePage) CommitList(listInfo *pipeline.TargetCommittedListInfo, NumberOfBlocks int, ...) (msg string, err error)
  • func (t *AzurePage) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)
  • func (t *AzurePage) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)
  • func (t *AzurePage) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)
• type MultiFile
  • func (t *MultiFile) CommitList(listInfo *pipeline.TargetCommittedListInfo, numberOfBlocks int, ...) (msg string, err error)
  • func (t *MultiFile) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)
  • func (t *MultiFile) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)
  • func (t *MultiFile) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)

Constants

const PageSize int64 = 512

PageSize size of page in Azure Page Blob storage

Functions

func NewAzureBlock

func NewAzureBlock(accountName string, accountKey string, container string) pipeline.TargetPipeline

NewAzureBlock creates a new Azure Block target

func NewAzurePage

func NewAzurePage(accountName string, accountKey string, container string) pipeline.TargetPipeline

NewAzurePage creates a new Azure Block target

func NewMultiFile

func NewMultiFile(overwrite bool, numberOfHandles int) pipeline.TargetPipeline

NewMultiFile creates a new multi file target and 'n' number of handles for concurrent writes to a file.

Types

type AzureBlock

type AzureBlock struct {
	Creds         *pipeline.StorageAccountCredentials
	Container     string
	StorageClient storage.BlobStorageClient
}

AzureBlock represents an Azure Block target

func (*AzureBlock) CommitList

func (t *AzureBlock) CommitList(listInfo *pipeline.TargetCommittedListInfo, numberOfBlocks int, targetName string) (msg string, err error)

CommitList implements CommitList from the pipeline.TargetPipeline interface. Commits the list of blocks to Azure Storage to finalize the transfer.

func (*AzureBlock) PreProcessSourceInfo

func (t *AzureBlock) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)

PreProcessSourceInfo implementation of PreProcessSourceInfo from the pipeline.TargetPipeline interface. Checks if uncommitted blocks are present and cleans them by creating an empty blob.

func (*AzureBlock) ProcessWrittenPart

func (t *AzureBlock) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)

ProcessWrittenPart implements ProcessWrittenPart from the pipeline.TargetPipeline interface. Appends the written part to a list. If the part is duplicated the list is updated with a reference, to the first occurrence of the block. If the first occurrence has not yet being processed, the part is requested to be placed back in the results channel (requeue == true).

func (*AzureBlock) WritePart

func (t *AzureBlock) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)

WritePart implements WritePart from the pipeline.TargetPipeline interface. Performs a PUT block operation with the data contained in the part.

type AzurePage

type AzurePage struct {
	Creds         *pipeline.StorageAccountCredentials
	Container     string
	StorageClient *storage.BlobStorageClient
}

AzurePage represents an Azure Block target

func (*AzurePage) CommitList

func (t *AzurePage) CommitList(listInfo *pipeline.TargetCommittedListInfo, NumberOfBlocks int, targetName string) (msg string, err error)

CommitList implements CommitList from the pipeline.TargetPipeline interface. Passthrough no need to a commit for page blob.

func (*AzurePage) PreProcessSourceInfo

func (t *AzurePage) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)

PreProcessSourceInfo implementation of PreProcessSourceInfo from the pipeline.TargetPipeline interface. initializes the page blob.

func (*AzurePage) ProcessWrittenPart

func (t *AzurePage) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)

ProcessWrittenPart implements ProcessWrittenPart from the pipeline.TargetPipeline interface. Passthrough no need to process a written part when transferring to a page blob.

func (*AzurePage) WritePart

func (t *AzurePage) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)

WritePart implements WritePart from the pipeline.TargetPipeline interface. Performs a PUT page operation with the data contained in the part. This assumes the part.BytesToRead is a multiple of the PageSize

type MultiFile

type MultiFile struct {
	Container       string
	FileHandles     map[string]chan *os.File
	NumberOfHandles int
	OverWrite       bool
	sync.Mutex
}

MultiFile represents an OS file(s) target

func (*MultiFile) CommitList

func (t *MultiFile) CommitList(listInfo *pipeline.TargetCommittedListInfo, numberOfBlocks int, targetName string) (msg string, err error)

CommitList implements CommitList from the pipeline.TargetPipeline interface. For a file download a final commit is not required and this implementation closes all the filehandles.

func (*MultiFile) PreProcessSourceInfo

func (t *MultiFile) PreProcessSourceInfo(source *pipeline.SourceInfo) (err error)

PreProcessSourceInfo implementation of PreProcessSourceInfo from the pipeline.TargetPipeline interface. Passthrough no need to pre-process for a file target.

func (*MultiFile) ProcessWrittenPart

func (t *MultiFile) ProcessWrittenPart(result *pipeline.WorkerResult, listInfo *pipeline.TargetCommittedListInfo) (requeue bool, err error)

ProcessWrittenPart implements ProcessWrittenPart from the pipeline.TargetPipeline interface. Passthrough implementation as no post-written-processing is required (e.g. maintain a list) when files are downloaded.

func (*MultiFile) WritePart

func (t *MultiFile) WritePart(part *pipeline.Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)

WritePart implements WritePart from the pipeline.TargetPipeline interface. Writes to a local file using a filehandle received from a channel.