common

package
v10.0.2+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 28, 2018 License: MIT Imports: 34 Imported by: 19

Documentation

Index

Constants

View Source 
const (
	AZCOPY_PATH_SEPARATOR_STRING = "/"
	AZCOPY_PATH_SEPARATOR_CHAR   = '/'
	OS_PATH_SEPARATOR            = string(os.PathSeparator)
)
View Source 
const (
	DefaultBlockBlobBlockSize = 8 * 1024 * 1024
	MaxBlockBlobBlockSize     = 100 * 1024 * 1024
	DefaultPageBlobChunkSize  = 4 * 1024 * 1024
	DefaultAzureFileChunkSize = 4 * 1024 * 1024
	MaxNumberOfBlocksPerBlob  = 50000
)

//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

View Source 
const ApplicationID = "579a7132-0e58-4d80-b1e1-7a1e2d337859"

ApplicationID represents 1st party ApplicationID for AzCopy. const ApplicationID = "a45c21f4-7066-40b4-97d8-14f4313c3caa" // 3rd party test ApplicationID for AzCopy.

View Source 
const AzcopyVersion = "10.0.2-Preview"
View Source 
const DefaultActiveDirectoryEndpoint = "https://login.microsoftonline.com"
View Source 
const DefaultTenantID = "microsoft.com"
View Source 
const EnvVarOAuthTokenInfo = "AZCOPY_OAUTH_TOKEN_INFO"

EnvVarOAuthTokenInfo passes oauth token info into AzCopy through environment variable. Note: this is only used for testing, and not encouraged to be used in production environments.

View Source 
const ErrorCodeEnvVarOAuthTokenInfoNotSet = "environment variable AZCOPY_OAUTH_TOKEN_INFO is not set"

ErrorCodeEnvVarOAuthTokenInfoNotSet defines error code when environment variable AZCOPY_OAUTH_TOKEN_INFO is not set.

View Source 
const IMDSAPIVersion = "2018-02-01"
View Source 
const MSIEndpoint = "http://169.254.169.254/metadata/identity/oauth2/token"
View Source 
const Resource = "https://storage.azure.com"

Resource used in azure storage OAuth authentication

View Source 
const UserAgent = "AzCopy/v" + AzcopyVersion

Variables

View Source 
var DefaultTokenExpiryWithinThreshold = time.Minute * 10
View Source 
var EBlockBlobTier = BlockBlobTier(0)
View Source 
var ECredentialType = CredentialType(0)
View Source 
var EExitCode = ExitCode(0)
View Source 
var EFromTo = FromTo(0)
View Source 
var EJobPriority = JobPriority(0)
View Source 
var EJobStatus = JobStatus(0)
View Source 
var ELocation = Location(0)
View Source 
var ELogLevel = LogLevel(pipeline.LogNone)
View Source 
var EOutputFormat = OutputFormat(0)
View Source 
var EPageBlobTier = PageBlobTier(0)
View Source 
var ERpcCmd = RpcCmd("")
View Source 
var ETransferStatus = TransferStatus(0)
View Source 
var GlobalTestOAuthInjection = TestOAuthInjection{
	DoTokenRefreshInjection: false,
	TokenRefreshDuration:    time.Second * 10,
}

GlobalTestOAuthInjection is the global setting for OAuth testing injection control

Functions

func AtomicMorphInt32 

func AtomicMorphInt32(target *int32, morpher AtomicMorpherInt32) interface{}

AtomicMorph atomically morphs target in to new value (and result) as indicated by the AtomicMorpher callback function.

func AtomicMorphInt64 

func AtomicMorphInt64(target *int64, morpher AtomicMorpherInt64) interface{}

AtomicMorph atomically morphs target in to new value (and result) as indicated bythe AtomicMorpher callback function.

func AtomicMorphUint32 

func AtomicMorphUint32(target *uint32, morpher AtomicMorpherUint32) interface{}

AtomicMorph atomically morphs target in to new value (and result) as indicated bythe AtomicMorpher callback function.

func AtomicMorphUint64 

func AtomicMorphUint64(target *uint64, morpher AtomicMorpherUint64) interface{}

AtomicMorph atomically morphs target in to new value (and result) as indicated bythe AtomicMorpher callback function.

func CreateBlobCredential 

func CreateBlobCredential(ctx context.Context, credInfo CredentialInfo, options CredentialOpOptions) azblob.Credential

CreateBlobCredential creates Blob credential according to credential info.

func CreateBlobFSCredential 

func CreateBlobFSCredential(ctx context.Context, credInfo CredentialInfo, options CredentialOpOptions) azbfs.Credential

CreateBlobFSCredential creates BlobFS credential according to credential info.

func CreateFileOfSize 

func CreateFileOfSize(destinationPath string, fileSize int64) (*os.File, error)

create a file, given its path and length

func CreateParentDirectoryIfNotExist 

func CreateParentDirectoryIfNotExist(destinationPath string) error

func EnvVarOAuthTokenInfoExists 

func EnvVarOAuthTokenInfoExists() bool

EnvVarOAuthTokenInfoExists verifies if environment variable for OAuthTokenInfo is specified. The method returns true if the environment variable is set. Note: This is useful for only checking whether the env var exists, please use GetTokenInfoFromEnvVar directly in the case getting token info is necessary.

func GetBlocksRoundedUp 

func GetBlocksRoundedUp(size uint64, blockSize uint64) uint16

GetBlocksRoundedUp returns the number of blocks given sie, rounded up

func IffError 

func IffError(test bool, trueVal, falseVal error) error

inline if functions

func IffString 

func IffString(test bool, trueVal, falseVal string) string

func IffUint8 

func IffUint8(test bool, trueVal, falseVal uint8) byte

func Iffint16 

func Iffint16(test bool, trueVal, falseVal int16) int16

func Iffint32 

func Iffint32(test bool, trueVal, falseVal int32) int32

func Iffint64 

func Iffint64(test bool, trueVal, falseVal int64) int64

func Iffint8 

func Iffint8(test bool, trueVal, falseVal int8) int8

func Iffloat64 

func Iffloat64(test bool, trueVal, falseVal float64) float64

func Iffuint16 

func Iffuint16(test bool, trueVal, falseVal uint16) uint16

func Iffuint32 

func Iffuint32(test bool, trueVal, falseVal uint32) uint32

func Iffuint64 

func Iffuint64(test bool, trueVal, falseVal uint64) uint64

func IsErrorEnvVarOAuthTokenInfoNotSet 

func IsErrorEnvVarOAuthTokenInfoNotSet(err error) bool

IsErrorEnvVarOAuthTokenInfoNotSet verifies if an error indicates environment variable AZCOPY_OAUTH_TOKEN_INFO is not set.

func PanicIfErr 

func PanicIfErr(err error)

captures the common logic of exiting if there's an expected error

Types

type AtomicMorpherInt32 

type AtomicMorpherInt32 func(startVal int32) (val int32, morphResult interface{})

AtomicMorpherInt32 identifies a method passed to and invoked by the AtomicMorphInt32 function. The AtomicMorpher callback is passed a startValue and based on this value it returns what the new value should be and the result that AtomicMorph should return to its caller.

type AtomicMorpherInt64 

type AtomicMorpherInt64 func(startVal int64) (val int64, morphResult interface{})

AtomicMorpherUint64 identifies a method passed to and invoked by the AtomicMorphUint64 function. The AtomicMorpher callback is passed a startValue and based on this value it returns what the new value should be and the result that AtomicMorph should return to its caller.

type AtomicMorpherUint32 

type AtomicMorpherUint32 func(startVal uint32) (val uint32, morphResult interface{})

AtomicMorpherUint32 identifies a method passed to and invoked by the AtomicMorph function. The AtomicMorpher callback is passed a startValue and based on this value it returns what the new value should be and the result that AtomicMorph should return to its caller.

type AtomicMorpherUint64 

type AtomicMorpherUint64 func(startVal uint64) (val uint64, morphResult interface{})

AtomicMorpherUint64 identifies a method passed to and invoked by the AtomicMorphUint64 function. The AtomicMorpher callback is passed a startValue and based on this value it returns what the new value should be and the result that AtomicMorph should return to its caller.

type BlobTransferAttributes 

type BlobTransferAttributes struct {
	//BlobType                 BlobType // The type of a blob - BlockBlob, PageBlob, AppendBlob
	ContentType              string        //The content type specified for the blob.
	ContentEncoding          string        //Specifies which content encodings have been applied to the blob.
	BlockBlobTier            BlockBlobTier // Specifies the tier to set on the block blobs.
	PageBlobTier             PageBlobTier  // Specifies the tier to set on the page blobs.
	Metadata                 string        //User-defined name-value pairs associated with the blob
	NoGuessMimeType          bool          // represents user decision to interpret the content-encoding from source file
	PreserveLastModifiedTime bool          // when downloading, tell engine to set file's timestamp to timestamp of blob
	BlockSizeInBytes         uint32
}

This struct represents the optional attribute for blob request header

type BlockBlobTier 

type BlockBlobTier uint8

func (BlockBlobTier) Archive 

func (BlockBlobTier) Archive() BlockBlobTier

func (BlockBlobTier) Cold 

func (BlockBlobTier) Cold() BlockBlobTier

func (BlockBlobTier) Cool 

func (BlockBlobTier) Cool() BlockBlobTier

func (BlockBlobTier) Hot 

func (BlockBlobTier) Hot() BlockBlobTier

func (BlockBlobTier) MarshalJSON 

func (bbt BlockBlobTier) MarshalJSON() ([]byte, error)

func (BlockBlobTier) None 

func (BlockBlobTier) None() BlockBlobTier

func (*BlockBlobTier) Parse 

func (bbt *BlockBlobTier) Parse(s string) error

func (BlockBlobTier) String 

func (bbt BlockBlobTier) String() string

func (BlockBlobTier) ToAccessTierType 

func (bbt BlockBlobTier) ToAccessTierType() azblob.AccessTierType

func (*BlockBlobTier) UnmarshalJSON 

func (bbt *BlockBlobTier) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type BlockBlobTier.

type ByteSlice 

type ByteSlice []byte

///////////////////////////////////////////////////////////////////////////////////////////////

type ByteSliceExtension 

type ByteSliceExtension struct {
	ByteSlice
}

func (ByteSliceExtension) RemoveBOM 

func (bs ByteSliceExtension) RemoveBOM() []byte

RemoveBOM removes any BOM from the byte slice

type CancelPauseResumeResponse 

type CancelPauseResumeResponse struct {
	ErrorMsg              string
	CancelledPauseResumed bool
}

type CopyJobPartOrderRequest 

type CopyJobPartOrderRequest struct {
	Version        Version     // version of the azcopy
	JobID          JobID       // Guid - job identifier
	PartNum        PartNumber  // part number of the job
	IsFinalPart    bool        // to determine the final part for a specific job
	ForceWrite     bool        // to determine if the existing needs to be overwritten or not. If set to true, existing blobs are overwritten
	Priority       JobPriority // priority of the task
	FromTo         FromTo
	Include        map[string]int
	Exclude        map[string]int
	Transfers      []CopyTransfer
	LogLevel       LogLevel
	BlobAttributes BlobTransferAttributes
	SourceSAS      string
	DestinationSAS string
	// commandString hold the user given command which is logged to the Job log file
	CommandString  string
	CredentialInfo CredentialInfo
}

This struct represents the job info (a single part) to be sent to the storage engine

type CopyJobPartOrderResponse 

type CopyJobPartOrderResponse struct {
	ErrorMsg   string
	JobStarted bool
}

type CopyTransfer 

type CopyTransfer struct {
	Source           string
	Destination      string
	LastModifiedTime time.Time //represents the last modified time of source which ensures that source hasn't changed while transferring
	SourceSize       int64     // size of the source entity in bytes.

	// Properties for service to service copy
	ContentType        string
	ContentEncoding    string
	ContentDisposition string
	ContentLanguage    string
	CacheControl       string
	ContentMD5         []byte
	Metadata           Metadata

	// Properties for blob copy only
	BlobType azblob.BlobType
}

This struct represent a single transfer entry with source and destination details

type CountPerSecond 

type CountPerSecond interface {
	// Add atomically adds delta to *addr and returns the new value.
	// To subtract a signed positive constant value c, do Add(^uint64(c-1)).
	Add(delta uint64) uint64 // Pass 0 to get the current count value
	LatestRate() float64
	Reset()
}

CountPerSecond ...

func NewCountPerSecond 

func NewCountPerSecond() CountPerSecond

type CredCache 

type CredCache struct {
	// contains filtered or unexported fields
}

CredCache manages credential caches. Use keyring in Linux OS. Session keyring is choosed, the session hooks key should be created since user first login (i.e. by pam). So the session is inherited by processes created from login session. When user logout, the session keyring is recycled.

func NewCredCache 

func NewCredCache(state string) *CredCache

NewCredCache creates a cred cache.

func (*CredCache) HasCachedToken 

func (c *CredCache) HasCachedToken() (bool, error)

HasCachedToken returns if there is cached token for current executing user.

func (*CredCache) LoadToken 

func (c *CredCache) LoadToken() (*OAuthTokenInfo, error)

LoadToken gets the cached oauth token.

func (*CredCache) RemoveCachedToken 

func (c *CredCache) RemoveCachedToken() error

RemoveCachedToken deletes the cached token.

func (*CredCache) SaveToken 

func (c *CredCache) SaveToken(token OAuthTokenInfo) error

SaveToken saves an oauth token.

type CredentialInfo 

type CredentialInfo struct {
	CredentialType CredentialType
	OAuthTokenInfo OAuthTokenInfo
}

CredentialInfo contains essential credential info which need be transited between modules, and used during creating Azure storage client Credential.

type CredentialOpOptions 

type CredentialOpOptions struct {
	LogInfo  func(string)
	LogError func(string)
	Panic    func(error)
	CallerID string

	// Used to cancel operations, if fatal error happend during operation.
	Cancel context.CancelFunc
}

CredentialOpOptions contains credential operations' parameters.

type CredentialType 

type CredentialType uint8

CredentialType defines the different types of credentials

func (CredentialType) Anonymous 

func (CredentialType) Anonymous() CredentialType

func (CredentialType) OAuthToken 

func (CredentialType) OAuthToken() CredentialType

func (*CredentialType) Parse 

func (ct *CredentialType) Parse(s string) error

func (CredentialType) SharedKey 

func (CredentialType) SharedKey() CredentialType

func (CredentialType) String 

func (ct CredentialType) String() string

func (CredentialType) Unknown 

func (CredentialType) Unknown() CredentialType

type ExitCode 

type ExitCode uint32

func (ExitCode) Error 

func (ExitCode) Error() ExitCode

func (ExitCode) Success 

func (ExitCode) Success() ExitCode

type FileURLPartsExtension 

type FileURLPartsExtension struct {
	azfile.FileURLParts
}

///////////////////////////////////////////////////////////////////////////////////////////////

func (FileURLPartsExtension) GetServiceURL 

func (parts FileURLPartsExtension) GetServiceURL() url.URL

func (FileURLPartsExtension) GetShareURL 

func (parts FileURLPartsExtension) GetShareURL() url.URL

type FromTo 

type FromTo uint16

FromTo defines the different types of sources/destination location combinations FromTo is 16 bit where first 8 bit represents the from location and other 8 bits represents the to location

func (FromTo) BlobBlob 

func (FromTo) BlobBlob() FromTo

func (FromTo) BlobFSLocal 

func (FromTo) BlobFSLocal() FromTo

func (FromTo) BlobLocal 

func (FromTo) BlobLocal() FromTo

func (FromTo) BlobPipe 

func (FromTo) BlobPipe() FromTo

func (FromTo) BlobTrash 

func (FromTo) BlobTrash() FromTo

func (FromTo) FileBlob 

func (FromTo) FileBlob() FromTo

func (FromTo) FileLocal 

func (FromTo) FileLocal() FromTo

func (FromTo) FilePipe 

func (FromTo) FilePipe() FromTo

func (FromTo) FileTrash 

func (FromTo) FileTrash() FromTo

func (*FromTo) From 

func (ft *FromTo) From() Location

func (*FromTo) FromAndTo 

func (ft *FromTo) FromAndTo(s string) (srcLocation, dstLocation Location, err error)

func (FromTo) LocalBlob 

func (FromTo) LocalBlob() FromTo

func (FromTo) LocalBlobFS 

func (FromTo) LocalBlobFS() FromTo

func (FromTo) LocalFile 

func (FromTo) LocalFile() FromTo

func (*FromTo) Parse 

func (ft *FromTo) Parse(s string) error

func (FromTo) PipeBlob 

func (FromTo) PipeBlob() FromTo

func (FromTo) PipeFile 

func (FromTo) PipeFile() FromTo

func (FromTo) String 

func (ft FromTo) String() string

func (*FromTo) To 

func (ft *FromTo) To() Location

func (FromTo) Unknown 

func (FromTo) Unknown() FromTo

type GetJobFromToRequest 

type GetJobFromToRequest struct {
	JobID JobID
}

GetJobFromToRequest indicates request to get job's FromTo info from job part plan header

type GetJobFromToResponse 

type GetJobFromToResponse struct {
	ErrorMsg    string
	FromTo      FromTo
	Source      string
	Destination string
}

GetJobFromToResponse indicates response to get job's FromTo info.

type HTTPResponseExtension 

type HTTPResponseExtension struct {
	*http.Response
}

///////////////////////////////////////////////////////////////////////////////////////////////

func (HTTPResponseExtension) IsSuccessStatusCode 

func (r HTTPResponseExtension) IsSuccessStatusCode(successStatusCodes ...int) bool

IsSuccessStatusCode checks if response's status code is contained in specified success status codes.

type ILogger 

type ILogger interface {
	ShouldLog(level pipeline.LogLevel) bool
	Log(level pipeline.LogLevel, msg string)
	Panic(err error)
}

type ILoggerCloser 

type ILoggerCloser interface {
	ILogger
	CloseLog()
}

func NewAppLogger 

func NewAppLogger(minimumLevelToLog pipeline.LogLevel, logFileFolder string) ILoggerCloser

type ILoggerResetable 

type ILoggerResetable interface {
	OpenLog()
	MinimumLogLevel() pipeline.LogLevel
	ILoggerCloser
}

func NewJobLogger 

func NewJobLogger(jobID JobID, minimumLevelToLog LogLevel, appLogger ILogger, logFileFolder string) ILoggerResetable

type IdentityInfo 

type IdentityInfo struct {
	ClientID string `json:"_identity_client_id"`
	ObjectID string `json:"_identity_object_id"`
	MSIResID string `json:"_identity_msi_res_id"`
}

IdentityInfo contains info for MSI.

func (*IdentityInfo) Validate 

func (identityInfo *IdentityInfo) Validate() error

Validate validates identity info, at most only one of clientID, objectID or MSI resource ID could be set.

type JobID 

type JobID UUID

func NewJobID 

func NewJobID() JobID

func ParseJobID 

func ParseJobID(jobID string) (JobID, error)

func (JobID) IsEmpty 

func (j JobID) IsEmpty() bool

var EmptyJobId JobID = JobID{}

func (JobID) MarshalJSON 

func (j JobID) MarshalJSON() ([]byte, error)

Implementing MarshalJSON() method for type JobID

func (JobID) String 

func (j JobID) String() string

func (*JobID) UnmarshalJSON 

func (j *JobID) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type JobID

type JobIDDetails 

type JobIDDetails struct {
	JobId         JobID
	CommandString string
}

type JobPriority 

type JobPriority uint8

JobPriority defines the transfer priorities supported by the Storage Transfer Engine's channels The default priority is Normal

func (JobPriority) Low 

func (JobPriority) Low() JobPriority

func (JobPriority) Normal 

func (JobPriority) Normal() JobPriority

func (JobPriority) String 

func (jp JobPriority) String() string

type JobStatus 

type JobStatus uint32 // Must be 32-bit for atomic operations

JobStatus indicates the status of a Job; the default is InProgress.

func (*JobStatus) AtomicLoad 

func (j *JobStatus) AtomicLoad() JobStatus

func (*JobStatus) AtomicStore 

func (j *JobStatus) AtomicStore(newJobStatus JobStatus)

func (JobStatus) Cancelled 

func (JobStatus) Cancelled() JobStatus

func (JobStatus) Cancelling 

func (JobStatus) Cancelling() JobStatus

func (JobStatus) Completed 

func (JobStatus) Completed() JobStatus

func (JobStatus) InProgress 

func (JobStatus) InProgress() JobStatus

func (JobStatus) MarshalJSON 

func (j JobStatus) MarshalJSON() ([]byte, error)

Implementing MarshalJSON() method for type JobStatus

func (*JobStatus) Parse 

func (j *JobStatus) Parse(s string) error

func (JobStatus) Paused 

func (JobStatus) Paused() JobStatus

func (JobStatus) String 

func (js JobStatus) String() string

func (*JobStatus) UnmarshalJSON 

func (j *JobStatus) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type JobStatus

type LifecycleMgr 

type LifecycleMgr interface {
	Progress(string)                                // print on the same line over and over again, not allowed to float up
	Info(string)                                    // simple print, allowed to float up
	Prompt(string) string                           // ask the user a question(after erasing the progress), then return the response
	Exit(string, ExitCode)                          // exit after printing
	SurrenderControl()                              // give up control, this should never return
	InitiateProgressReporting(WorkController, bool) // start writing progress with another routine
}

create a public interface so that consumers outside of this package can refer to the lifecycle manager but they would not be able to instantiate one

func GetLifecycleMgr 

func GetLifecycleMgr() LifecycleMgr

type ListContainerResponse 

type ListContainerResponse struct {
	Blobs []string
}

ListContainerResponse represents the list of blobs within the container.

type ListJobSummaryResponse 

type ListJobSummaryResponse struct {
	ErrorMsg  string
	Timestamp time.Time
	JobID     JobID
	// TODO: added for debugging purpose. remove later
	ActiveConnections int64
	// CompleteJobOrdered determines whether the Job has been completely ordered or not
	CompleteJobOrdered    bool
	JobStatus             JobStatus
	TotalTransfers        uint32
	TransfersCompleted    uint32
	TransfersFailed       uint32
	TransfersSkipped      uint32
	BytesOverWire         uint64
	TotalBytesTransferred uint64
	FailedTransfers       []TransferDetail
	SkippedTransfers      []TransferDetail
}

represents the JobProgressPercentage Summary response for list command when requested the Job Progress Summary for given JobId

type ListJobTransfersRequest 

type ListJobTransfersRequest struct {
	JobID    JobID
	OfStatus TransferStatus
}

type ListJobTransfersResponse 

type ListJobTransfersResponse struct {
	ErrorMsg string
	JobID    JobID
	Details  []TransferDetail
}

represents the list of Details and details of number of transfers

type ListJobsResponse 

type ListJobsResponse struct {
	ErrorMessage string
	JobIDDetails []JobIDDetails
}

ListJobsResponse represent the Job with JobId and

type ListRequest 

type ListRequest struct {
	JobID    JobID
	OfStatus string // TODO: OfStatus with string type sounds not good, change it to enum
	Output   OutputFormat
}

represents the raw list command input from the user when requested the list of transfer with given status for given JobId

type Location 

type Location uint8

Location indicates the type of Location

func (Location) Blob 

func (Location) Blob() Location

func (Location) BlobFS 

func (Location) BlobFS() Location

func (Location) File 

func (Location) File() Location

func (Location) Local 

func (Location) Local() Location

func (Location) Pipe 

func (Location) Pipe() Location

func (Location) String 

func (l Location) String() string

func (Location) Unknown 

func (Location) Unknown() Location

type LogLevel 

type LogLevel uint8

func (LogLevel) Debug 

func (LogLevel) Debug() LogLevel

func (LogLevel) Error 

func (LogLevel) Error() LogLevel

func (LogLevel) Fatal 

func (LogLevel) Fatal() LogLevel

func (LogLevel) Info 

func (LogLevel) Info() LogLevel

func (LogLevel) None 

func (LogLevel) None() LogLevel

func (LogLevel) Panic 

func (LogLevel) Panic() LogLevel

func (*LogLevel) Parse 

func (ll *LogLevel) Parse(s string) error

func (LogLevel) String 

func (ll LogLevel) String() string

func (LogLevel) ToPipelineLogLevel 

func (ll LogLevel) ToPipelineLogLevel() pipeline.LogLevel

func (LogLevel) Warning 

func (LogLevel) Warning() LogLevel

type MMF 

type MMF struct {
	// contains filtered or unexported fields
}

func NewMMF 

func NewMMF(file *os.File, writable bool, offset int64, length int64) (*MMF, error)

func (*MMF) Slice 

func (m *MMF) Slice() []byte

Slice() returns the memory mapped byte slice

func (*MMF) Unmap 

func (m *MMF) Unmap()

To unmap, we need exclusive (write) access to the MMF and then we set isMapped to false so that future readers know the MMF is unusable.

func (*MMF) UnuseMMF 

func (m *MMF) UnuseMMF()

RUnlock unlocks the held lock

func (*MMF) UseMMF 

func (m *MMF) UseMMF() bool

type Metadata 

type Metadata map[string]string

Metadata used in AzCopy.

func FromAzBlobMetadataToCommonMetadata 

func FromAzBlobMetadataToCommonMetadata(m azblob.Metadata) Metadata

FromAzBlobMetadataToCommonMetadata converts azblob's metadata to common metadata.

func FromAzFileMetadataToCommonMetadata 

func FromAzFileMetadataToCommonMetadata(m azfile.Metadata) Metadata

FromAzFileMetadataToCommonMetadata converts azfile's metadata to common metadata.

func UnMarshalToCommonMetadata 

func UnMarshalToCommonMetadata(metadataString string) (Metadata, error)

UnMarshalToCommonMetadata unmarshals string to common metadata.

func (Metadata) Marshal 

func (m Metadata) Marshal() (string, error)

Marshal marshals metadata to string.

func (Metadata) ToAzBlobMetadata 

func (m Metadata) ToAzBlobMetadata() azblob.Metadata

ToAzBlobMetadata converts metadata to azblob's metadata.

func (Metadata) ToAzFileMetadata 

func (m Metadata) ToAzFileMetadata() azfile.Metadata

ToAzFileMetadata converts metadata to azfile's metadata.

type NoCopy 

type NoCopy struct {
	// contains filtered or unexported fields
}

The NoCopy struct is used as a field inside another struct that should not be copied by value. After embedded this field, the out struct's members can call the Check method which will panic if it detects the out struct has been copied by value.

func (*NoCopy) Check 

func (nc *NoCopy) Check()

Check panics if the struct embedded this NoCopy field has been copied by value.

type OAuthTokenInfo 

type OAuthTokenInfo struct {
	adal.Token
	Tenant                  string `json:"_tenant"`
	ActiveDirectoryEndpoint string `json:"_ad_endpoint"`
	Identity                bool   `json:"_identity"`
	IdentityInfo            IdentityInfo
}

OAuthTokenInfo contains info necessary for refresh OAuth credentials.

func JSONToTokenInfo 

func JSONToTokenInfo(b []byte) (*OAuthTokenInfo, error)

JSONToTokenInfo converts bytes to OAuthTokenInfo

func (*OAuthTokenInfo) GetNewTokenFromMSI 

func (credInfo *OAuthTokenInfo) GetNewTokenFromMSI(ctx context.Context) (*adal.Token, error)

GetNewTokenFromMSI get token from Azure Instance Metadata Service identity endpoint. For details, please refer to https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

func (OAuthTokenInfo) IsEmpty 

func (credInfo OAuthTokenInfo) IsEmpty() bool

IsEmpty returns if current OAuthTokenInfo is empty and doesn't contain any useful info.

func (*OAuthTokenInfo) Refresh 

func (credInfo *OAuthTokenInfo) Refresh(ctx context.Context) (*adal.Token, error)

Refresh gets new token with token info.

func (*OAuthTokenInfo) RefreshTokenWithUserCredential 

func (credInfo *OAuthTokenInfo) RefreshTokenWithUserCredential(ctx context.Context) (*adal.Token, error)

RefreshTokenWithUserCredential get new token with user credential through refresh.

func (OAuthTokenInfo) ToJSON 

func (credInfo OAuthTokenInfo) ToJSON() ([]byte, error)

ToJSON converts OAuthTokenInfo to json format.

type OutputFormat 

type OutputFormat uint32

func (OutputFormat) Json 

func (OutputFormat) Json() OutputFormat

func (OutputFormat) None 

func (OutputFormat) None() OutputFormat

func (*OutputFormat) Parse 

func (of *OutputFormat) Parse(s string) error

func (OutputFormat) Text 

func (OutputFormat) Text() OutputFormat

type PageBlobTier 

type PageBlobTier uint8

func (PageBlobTier) MarshalJSON 

func (pbt PageBlobTier) MarshalJSON() ([]byte, error)

func (PageBlobTier) None 

func (PageBlobTier) None() PageBlobTier

func (PageBlobTier) P10 

func (PageBlobTier) P10() PageBlobTier

func (PageBlobTier) P20 

func (PageBlobTier) P20() PageBlobTier

func (PageBlobTier) P30 

func (PageBlobTier) P30() PageBlobTier

func (PageBlobTier) P4 

func (PageBlobTier) P4() PageBlobTier

func (PageBlobTier) P40 

func (PageBlobTier) P40() PageBlobTier

func (PageBlobTier) P50 

func (PageBlobTier) P50() PageBlobTier

func (PageBlobTier) P6 

func (PageBlobTier) P6() PageBlobTier

func (*PageBlobTier) Parse 

func (pbt *PageBlobTier) Parse(s string) error

func (PageBlobTier) String 

func (pbt PageBlobTier) String() string

func (PageBlobTier) ToAccessTierType 

func (pbt PageBlobTier) ToAccessTierType() azblob.AccessTierType

func (*PageBlobTier) UnmarshalJSON 

func (pbt *PageBlobTier) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type BlockBlobTier.

type PartNumber 

type PartNumber uint32

type ResumeJobRequest 

type ResumeJobRequest struct {
	JobID           JobID
	SourceSAS       string
	DestinationSAS  string
	IncludeTransfer map[string]int
	ExcludeTransfer map[string]int
	CredentialInfo  CredentialInfo
}

type RpcCmd 

type RpcCmd string

JobStatus indicates the status of a Job; the default is InProgress.

func (RpcCmd) CancelJob 

func (RpcCmd) CancelJob() RpcCmd

func (RpcCmd) CopyJobPartOrder 

func (RpcCmd) CopyJobPartOrder() RpcCmd

func (RpcCmd) GetJobFromTo 

func (RpcCmd) GetJobFromTo() RpcCmd

func (RpcCmd) ListJobSummary 

func (RpcCmd) ListJobSummary() RpcCmd

func (RpcCmd) ListJobTransfers 

func (RpcCmd) ListJobTransfers() RpcCmd

func (RpcCmd) ListJobs 

func (RpcCmd) ListJobs() RpcCmd

func (RpcCmd) None 

func (RpcCmd) None() RpcCmd

func (*RpcCmd) Parse 

func (c *RpcCmd) Parse(s string) error

func (RpcCmd) Pattern 

func (c RpcCmd) Pattern() string

func (RpcCmd) PauseJob 

func (RpcCmd) PauseJob() RpcCmd

func (RpcCmd) ResumeJob 

func (RpcCmd) ResumeJob() RpcCmd

func (RpcCmd) String 

func (c RpcCmd) String() string

type Status 

type Status uint32

type SyncJobPartOrderRequest 

type SyncJobPartOrderRequest struct {
	JobID            JobID
	FromTo           FromTo
	PartNumber       PartNumber
	LogLevel         LogLevel
	Include          map[string]int
	Exclude          map[string]int
	BlockSizeInBytes uint32
	SourceSAS        string
	DestinationSAS   string
	CopyJobRequest   CopyJobPartOrderRequest
	DeleteJobRequest CopyJobPartOrderRequest
	// FilesDeletedLocally is used to keep track of the file that are deleted locally
	// Since local files to delete are not sent as transfer to STE
	// the count of the local files deletion is tracked using it.
	FilesDeletedLocally int
	// commandString hold the user given command which is logged to the Job log file
	CommandString  string
	CredentialInfo CredentialInfo
}

type TestOAuthInjection 

type TestOAuthInjection struct {
	DoTokenRefreshInjection bool
	TokenRefreshDuration    time.Duration
}

TestOAuthInjection controls variables for OAuth testing injections

type TransferDetail 

type TransferDetail struct {
	Src            string
	Dst            string
	TransferStatus TransferStatus
	ErrorCode      int32
}

represents the Details and details of a single transfer

type TransferStatus 

type TransferStatus int32 // Must be 32-bit for atomic operations; negative #s represent a specific failure code

func (TransferStatus) All 

func (TransferStatus) All() TransferStatus

Transfer is any of the three possible state (InProgress, Completer or Failed)

func (*TransferStatus) AtomicLoad 

func (ts *TransferStatus) AtomicLoad() TransferStatus

func (*TransferStatus) AtomicStore 

func (ts *TransferStatus) AtomicStore(newTransferStatus TransferStatus)

func (TransferStatus) BlobAlreadyExistsFailure 

func (TransferStatus) BlobAlreadyExistsFailure() TransferStatus

func (TransferStatus) BlobTierFailure 

func (TransferStatus) BlobTierFailure() TransferStatus

Transfer failed due to failure while Setting blob tier.

func (TransferStatus) DidFail 

func (ts TransferStatus) DidFail() bool

func (TransferStatus) Failed 

func (TransferStatus) Failed() TransferStatus

Transfer failed due to some error. This status does represent the state when transfer is cancelled

func (TransferStatus) FileAlreadyExistsFailure 

func (TransferStatus) FileAlreadyExistsFailure() TransferStatus

func (TransferStatus) MarshalJSON 

func (ts TransferStatus) MarshalJSON() ([]byte, error)

Implementing MarshalJSON() method for type Transfer Status

func (TransferStatus) NotStarted 

func (TransferStatus) NotStarted() TransferStatus

Transfer is ready to transfer and not started transferring yet

func (*TransferStatus) Parse 

func (ts *TransferStatus) Parse(s string) error

func (TransferStatus) ShouldTransfer 

func (ts TransferStatus) ShouldTransfer() bool

func (TransferStatus) Started 

func (TransferStatus) Started() TransferStatus

Transfer started & at least 1 chunk has successfully been transfered. Used to resume a transfer that started to avoid transfering all chunks thereby improving performance

func (TransferStatus) String 

func (ts TransferStatus) String() string

func (TransferStatus) Success 

func (TransferStatus) Success() TransferStatus

Transfer successfully completed

func (*TransferStatus) UnmarshalJSON 

func (ts *TransferStatus) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type Transfer Status

type URLExtension 

type URLExtension struct {
	url.URL
}

///////////////////////////////////////////////////////////////////////////////////////////////

func (URLExtension) RedactSigQueryParamForLogging 

func (u URLExtension) RedactSigQueryParamForLogging() string

type URLStringExtension 

type URLStringExtension string

///////////////////////////////////////////////////////////////////////////////////////////////

func (URLStringExtension) RedactSigQueryParamForLogging 

func (s URLStringExtension) RedactSigQueryParamForLogging() string

type UUID 

type UUID struct {
	D1 uint32
	D2 uint16
	D3 uint16
	D4 [8]uint8
}

A UUID representation compliant with specification in RFC 4122 document.

func NewUUID 

func NewUUID() (u UUID)

NewUUID returns a new UUID using RFC 4122 algorithm.

func ParseUUID 

func ParseUUID(uuidStr string) (UUID, error)

ParseUUID parses a string formatted as "003020100-0504-0706-0809-0a0b0c0d0e0f" or "{03020100-0504-0706-0809-0a0b0c0d0e0f}" into a UUID.

func (UUID) MarshalJSON 

func (u UUID) MarshalJSON() ([]byte, error)

Implementing MarshalJSON() method for type UUID

func (UUID) String 

func (u UUID) String() string

String returns an unparsed version of the generated UUID sequence.

func (*UUID) UnmarshalJSON 

func (u *UUID) UnmarshalJSON(b []byte) error

Implementing UnmarshalJSON() method for type UUID

type UserOAuthTokenManager 

type UserOAuthTokenManager struct {
	// contains filtered or unexported fields
}

UserOAuthTokenManager for token management.

func NewUserOAuthTokenManagerInstance 

func NewUserOAuthTokenManagerInstance(userTokenCachePath string) *UserOAuthTokenManager

NewUserOAuthTokenManagerInstance creates a token manager instance.

func (*UserOAuthTokenManager) GetCachedTokenInfo 

func (uotm *UserOAuthTokenManager) GetCachedTokenInfo(ctx context.Context) (*OAuthTokenInfo, error)

GetCachedTokenInfo get a fresh token from local disk cache. If access token is expired, it will refresh the token. If refresh token is expired, the method will fail and return failure reason. Fresh token is persisted if acces token or refresh token is changed.

func (*UserOAuthTokenManager) GetTokenInfo 

func (uotm *UserOAuthTokenManager) GetTokenInfo(ctx context.Context) (*OAuthTokenInfo, error)

GetTokenInfo gets token info, it follows rule:

  1. If there is token passed from environment variable(note this is only for testing purpose), use token passed from environment variable.
  2. Otherwise, try to get token from cache.

This method either successfully return token, or return error.

func (*UserOAuthTokenManager) GetTokenInfoFromEnvVar 

func (uotm *UserOAuthTokenManager) GetTokenInfoFromEnvVar(ctx context.Context) (*OAuthTokenInfo, error)

GetTokenInfoFromEnvVar gets token info from environment variable.

func (*UserOAuthTokenManager) HasCachedToken 

func (uotm *UserOAuthTokenManager) HasCachedToken() (bool, error)

HasCachedToken returns if there is cached token in token manager.

func (*UserOAuthTokenManager) MSILogin 

func (uotm *UserOAuthTokenManager) MSILogin(ctx context.Context, identityInfo IdentityInfo, persist bool) (*OAuthTokenInfo, error)

MSILogin tries to get token from MSI, persist indicates whether to cache the token on local disk.

func (*UserOAuthTokenManager) RemoveCachedToken 

func (uotm *UserOAuthTokenManager) RemoveCachedToken() error

RemoveCachedToken delete all the cached token.

func (*UserOAuthTokenManager) UserLogin 

func (uotm *UserOAuthTokenManager) UserLogin(tenantID, activeDirectoryEndpoint string, persist bool) (*OAuthTokenInfo, error)

UserLogin interactively logins in with specified tenantID and activeDirectoryEndpoint, persist indicates whether to cache the token on local disk.

type Version 

type Version uint32

type WorkController 

type WorkController interface {
	Cancel(mgr LifecycleMgr)               // handle to cancel the work
	ReportProgressOrExit(mgr LifecycleMgr) // print the progress status, optionally exit the application if work is done
}

for the lifecycleMgr to babysit a job, it must be given a controller to get information about the job

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.