Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var MD5ToBlockID = make(map[string]int)
MD5ToBlockID simple lookup table mapping an MD5 string to a blockID
var MD5ToBlockIDLock sync.RWMutex
MD5ToBlockIDLock a lock for the map
Functions ¶
func NewBytesBufferChan ¶
NewBytesBufferChan creates a channel with 'n' slices of []bytes. 'n' is the bufferQCapacity. If the bufferQCapacity times bufferSize is greater than 1 GB 'n' is limited to a value that meets the constraint.
Types ¶
type Part ¶
type Part struct {
Offset uint64
BlockSize uint32
BytesToRead uint32
Data []byte // The data for the block. Can be nil if not yet read from the source
BlockID string
DuplicateOfBlockOrdinal int // -1 if not a duplicate of another, already read, block.
Ordinal int // sequentially assigned at creation time to enable chunk ordering (0,1,2)
SourceURI string
TargetAlias string
NumberOfBlocks int
BufferQ chan []byte
// contains filtered or unexported fields
}
Part description of and data for a block of the source
func ConstructPartsQueue ¶
func ConstructPartsQueue(size uint64, blockSize uint64, sourceURI string, targetAlias string, bufferQ chan []byte) (parts []Part, numOfBlocks int)
ConstructPartsQueue constructs a slice of parts calculated arithmetically from blockSize and size.
func NewPart ¶
func NewPart(offset uint64, bytesCount uint32, ordinal int, sourceURI string, targetAlias string) *Part
NewPart represents a block of data to be read from the source and written to the target
func (*Part) GetBuffer ¶
func (p *Part) GetBuffer()
GetBuffer sets the part's buffer (p.Data) to slice of bytes of size BytesToRead. The slice is read from channel of pre-allocated buffers. If the channel is empty a new slice is allocated.
func (*Part) IsMD5Computed ¶
IsMD5Computed true if the MD5 value was computed
func (*Part) LookupMD5DupeOrdinal ¶
LookupMD5DupeOrdinal finds the ordinal of a block which has the same data as this one. If none, then -1.
func (*Part) ReturnBuffer ¶
func (p *Part) ReturnBuffer()
ReturnBuffer adds part's buffer to channel so it can be reused.
type PartsPartition ¶
type PartsPartition struct {
Offset int64
NumOfParts int
TotalNumOfParts int64
TotalSize int64
PartitionSize int64
Parts []Part
}
PartsPartition represents a set of parts that can be read sequentially starting at the partition's Offset
func ConstructPartsPartition ¶
func ConstructPartsPartition(numOfPartitions int, size int64, blockSize int64, sourceURI string, targetAlias string, bufferQ chan []byte) []PartsPartition
ConstructPartsPartition creates a slice of PartsPartition with a len of numberOfPartitions.
type SourceInfo ¶
SourceInfo basic information about a source item.
type SourcePipeline ¶
type SourcePipeline interface {
ConstructBlockInfoQueue(blockSize uint64) (partitionQ chan PartsPartition, partsQ chan Part, numOfBlocks int, Size uint64)
ExecuteReader(partitionQ chan PartsPartition, partsQ chan Part, readPartsQ chan Part, id int, wg *sync.WaitGroup)
GetSourcesInfo() []SourceInfo
}
SourcePipeline operations that abstract the creation of the empty and read parts channels.
type StorageAccountCredentials ¶
type StorageAccountCredentials struct {
AccountName string // short name of the storage account. e.g., mystore
AccountKey string // Base64-encoded storage account key
}
StorageAccountCredentials a central location for account info.
type TargetCommittedListInfo ¶
type TargetCommittedListInfo struct {
List interface{}
}
TargetCommittedListInfo contains a list parts that have been written to a target.
type TargetPipeline ¶
type TargetPipeline interface {
PreProcessSourceInfo(source *SourceInfo, blockSize uint64) (err error)
CommitList(listInfo *TargetCommittedListInfo, numberOfBlocks int, targetName string) (msg string, err error)
WritePart(part *Part) (duration time.Duration, startTime time.Time, numOfRetries int, err error)
ProcessWrittenPart(result *WorkerResult, listInfo *TargetCommittedListInfo) (requeue bool, err error)
}
TargetPipeline operations that abstract how parts a written and processed to a given target
Source Files