importer

package
v1.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 14, 2019 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrInvalidPath = fmt.Errorf("invalid transfer path")

ErrInvalidPath indicates that the path is invalid.

View Source 
var ErrRequiresScratchSpace = fmt.Errorf("scratch space required and none found")

ErrRequiresScratchSpace indicates that we require scratch space.

Functions

func CleanDir added in v1.7.0 

func CleanDir(dest string) error

CleanDir cleans the contents of a directory including its sub directories, but does NOT remove the directory itself.

func ParseEndpoint 

func ParseEndpoint(endpt string) (*url.URL, error)

ParseEndpoint parses the required endpoint and return the url struct.

func ResizeImage added in v1.4.0 

func ResizeImage(dataFile, imageSize string, totalTargetSpace int64) error

ResizeImage resizes the images to match the requested size. Sometimes provisioners misbehave and the available space is not the same as the requested space. For those situations we compare the available space to the requested space and use the smallest of the two values.

Types

type DataProcessor added in v1.8.0 

type DataProcessor struct {
	// contains filtered or unexported fields
}

DataProcessor holds the fields needed to process data from a data provider.

func NewDataProcessor added in v1.8.0 

func NewDataProcessor(dataSource DataSourceInterface, dataFile, dataDir, scratchDataDir, requestImageSize string) *DataProcessor

NewDataProcessor create a new instance of a data processor using the passed in data provider.

func (*DataProcessor) ProcessData added in v1.8.0 

func (dp *DataProcessor) ProcessData() error

ProcessData is the main processing loop.

type DataSourceInterface added in v1.8.0 

type DataSourceInterface interface {
	// Info is called to get initial information about the data.
	Info() (ProcessingPhase, error)
	// Transfer is called to transfer the data from the source to the path passed in.
	Transfer(path string) (ProcessingPhase, error)
	// TransferFile is called to transfer the data from the source to the file passed in.
	TransferFile(fileName string) (ProcessingPhase, error)
	// Process is called to do any special processing before giving the url to the data back to the processor
	Process() (ProcessingPhase, error)
	// Geturl returns the url that the data processor can use when converting the data.
	GetURL() *url.URL
	// Close closes any readers or other open resources.
	Close() error
}

DataSourceInterface is the interface all data providers should implement.

type FormatReaders added in v1.8.0 

type FormatReaders struct {
	Convert  bool
	Archived bool
	// contains filtered or unexported fields
}

FormatReaders contains the stack of readers needed to get information from the input stream (io.ReadCloser)

func NewFormatReaders added in v1.8.0 

func NewFormatReaders(stream io.ReadCloser, total uint64) (*FormatReaders, error)

NewFormatReaders creates a new instance of FormatReaders using the input stream and content type passed in.

func (*FormatReaders) Close added in v1.8.0 

func (fr *FormatReaders) Close() (rtnerr error)

Close Readers in reverse order.

func (*FormatReaders) StartProgressUpdate added in v1.10.0 

func (fr *FormatReaders) StartProgressUpdate()

StartProgressUpdate starts the go routine to automatically update the progress on a set interval.

func (*FormatReaders) TopReader added in v1.8.0 

func (fr *FormatReaders) TopReader() io.ReadCloser

TopReader return the top-level io.ReadCloser from the receiver Reader "stack".

type HTTPDataSource added in v1.8.0 

type HTTPDataSource struct {
	// contains filtered or unexported fields
}

HTTPDataSource is the data provider for http(s) endpoints. Sequence of phases: 1a. Info -> Convert (In Info phase the format readers are configured), if the source Reader image is not archived, and no custom CA is used, and can be converted by QEMU-IMG (RAW/QCOW2) 1b. Info -> TransferArchive if the content type is archive 1c. Info -> Transfer in all other cases. 2a. Transfer -> Process if content type is kube virt 2b. Transfer -> Complete if content type is archive (Transfer is called with the target instead of the scratch space). Non block PVCs only. 3. Process -> Convert

func NewHTTPDataSource added in v1.8.0 

func NewHTTPDataSource(endpoint, accessKey, secKey, certDir string, contentType cdiv1.DataVolumeContentType) (*HTTPDataSource, error)

NewHTTPDataSource creates a new instance of the http data provider.

func (*HTTPDataSource) Close added in v1.8.0 

func (hs *HTTPDataSource) Close() error

Close all readers.

func (*HTTPDataSource) GetURL added in v1.8.0 

func (hs *HTTPDataSource) GetURL() *url.URL

GetURL returns the URI that the data processor can use when converting the data.

func (*HTTPDataSource) Info added in v1.8.0 

func (hs *HTTPDataSource) Info() (ProcessingPhase, error)

Info is called to get initial information about the data.

func (*HTTPDataSource) Process added in v1.8.0 

func (hs *HTTPDataSource) Process() (ProcessingPhase, error)

Process is called to do any special processing before giving the URI to the data back to the processor

func (*HTTPDataSource) Transfer added in v1.8.0 

func (hs *HTTPDataSource) Transfer(path string) (ProcessingPhase, error)

Transfer is called to transfer the data from the source to a scratch location.

func (*HTTPDataSource) TransferFile added in v1.8.0 

func (hs *HTTPDataSource) TransferFile(fileName string) (ProcessingPhase, error)

TransferFile is called to transfer the data from the source to the passed in file.

type ProcessingPhase added in v1.8.0 

type ProcessingPhase string

ProcessingPhase is the current phase being processed. 

const (
	// ProcessingPhaseInfo is the first phase, during this phase the source obtains information needed to determine which phase to go to next.
	ProcessingPhaseInfo ProcessingPhase = "Info"
	// ProcessingPhaseTransferScratch is the phase in which the data source writes data to the scratch space.
	ProcessingPhaseTransferScratch ProcessingPhase = "TransferScratch"
	// ProcessingPhaseTransferDataDir is the phase in which the data source writes data directly to the target path without conversion.
	ProcessingPhaseTransferDataDir ProcessingPhase = "TransferDataDir"
	// ProcessingPhaseTransferDataFile is the phase in which the data source writes data directly to the target file without conversion.
	ProcessingPhaseTransferDataFile ProcessingPhase = "TransferDataFile"
	// ProcessingPhaseProcess is the phase in which the data source processes the data just written to the scratch space.
	ProcessingPhaseProcess ProcessingPhase = "Process"
	// ProcessingPhaseConvert is the phase in which the data is taken from the url provided by the source, and it is converted to the target RAW disk image format.
	// The url can be an http end point or file system end point.
	ProcessingPhaseConvert ProcessingPhase = "Convert"
	// ProcessingPhaseResize the disk image, this is only needed when the target contains a file system (block device do not need a resize)
	ProcessingPhaseResize ProcessingPhase = "Resize"
	// ProcessingPhaseComplete is the phase where the entire process completed successfully and we can exit gracefully.
	ProcessingPhaseComplete ProcessingPhase = "Complete"
	// ProcessingPhaseError is the phase in which we encountered an error and need to exit ungracefully.
	ProcessingPhaseError ProcessingPhase = "Error"
)

type RegistryDataSource added in v1.8.0 

type RegistryDataSource struct {
	// contains filtered or unexported fields
}

RegistryDataSource is the struct containing the information needed to import from a registry data source. Sequence of phases: 1. Info -> Transfer 2. Transfer -> Process 3. Process -> Convert (In the process phase the container image layers are extracted, and the url is pointed to the file determined to be the disk image)

func NewRegistryDataSource added in v1.8.0 

func NewRegistryDataSource(endpoint, accessKey, secKey, certDir string, insecureTLS bool) *RegistryDataSource

NewRegistryDataSource creates a new instance of the Registry Data Source.

func (*RegistryDataSource) Close added in v1.8.0 

func (rd *RegistryDataSource) Close() error

Close closes any readers or other open resources.

func (*RegistryDataSource) GetURL added in v1.8.0 

func (rd *RegistryDataSource) GetURL() *url.URL

GetURL returns the url that the data processor can use when converting the data.

func (*RegistryDataSource) Info added in v1.8.0 

func (rd *RegistryDataSource) Info() (ProcessingPhase, error)

Info is called to get initial information about the data. No information available for registry currently.

func (*RegistryDataSource) Process added in v1.8.0 

func (rd *RegistryDataSource) Process() (ProcessingPhase, error)

Process is called to do any special processing before giving the url to the data back to the processor

func (*RegistryDataSource) Transfer added in v1.8.0 

func (rd *RegistryDataSource) Transfer(path string) (ProcessingPhase, error)

Transfer is called to transfer the data from the source registry to a temporary location.

func (*RegistryDataSource) TransferFile added in v1.8.0 

func (rd *RegistryDataSource) TransferFile(fileName string) (ProcessingPhase, error)

TransferFile is called to transfer the data from the source to the passed in file.

type S3Client added in v1.8.0 

type S3Client interface {
	GetObject(bucketName, objectName string, opts minio.GetObjectOptions) (*minio.Object, error)
}

S3Client is the interface to the used S3 client.

type S3DataSource added in v1.8.0 

type S3DataSource struct {
	// contains filtered or unexported fields
}

S3DataSource is the struct containing the information needed to import from an S3 data source. Sequence of phases: 1. Info -> Transfer 2. Transfer -> Process 3. Process -> Convert

func NewS3DataSource added in v1.8.0 

func NewS3DataSource(endpoint, accessKey, secKey string) (*S3DataSource, error)

NewS3DataSource creates a new instance of the S3DataSource

func (*S3DataSource) Close added in v1.8.0 

func (sd *S3DataSource) Close() error

Close closes any readers or other open resources.

func (*S3DataSource) GetURL added in v1.8.0 

func (sd *S3DataSource) GetURL() *url.URL

GetURL returns the url that the data processor can use when converting the data.

func (*S3DataSource) Info added in v1.8.0 

func (sd *S3DataSource) Info() (ProcessingPhase, error)

Info is called to get initial information about the data.

func (*S3DataSource) Process added in v1.8.0 

func (sd *S3DataSource) Process() (ProcessingPhase, error)

Process is called to do any special processing before giving the url to the data back to the processor

func (*S3DataSource) Transfer added in v1.8.0 

func (sd *S3DataSource) Transfer(path string) (ProcessingPhase, error)

Transfer is called to transfer the data from the source to a temporary location.

func (*S3DataSource) TransferFile added in v1.8.0 

func (sd *S3DataSource) TransferFile(fileName string) (ProcessingPhase, error)

TransferFile is called to transfer the data from the source to the passed in file.

type UploadDataSource added in v1.8.0 

type UploadDataSource struct {
	// contains filtered or unexported fields
}

UploadDataSource contains all the information need to upload data into a data volume. Sequence of phases: 1a. ProcessingPhaseInfo -> ProcessingPhaseTransferScratch (In Info phase the format readers are configured) In case the readers don't contain a raw file. 1b. ProcessingPhaseInfo -> ProcessingPhaseTransferDataFile, in the case the readers contain a raw file. 2a. ProcessingPhaseTransferScratch -> ProcessingPhaseProcess 2b. ProcessingPhaseTransferDataFile -> ProcessingPhaseResize 3. ProcessingPhaseProcess -> ProcessingPhaseConvert

func NewUploadDataSource added in v1.8.0 

func NewUploadDataSource(stream io.ReadCloser) *UploadDataSource

NewUploadDataSource creates a new instance of an UploadDataSource

func (*UploadDataSource) Close added in v1.8.0 

func (ud *UploadDataSource) Close() error

Close closes any readers or other open resources.

func (*UploadDataSource) GetURL added in v1.8.0 

func (ud *UploadDataSource) GetURL() *url.URL

GetURL returns the url that the data processor can use when converting the data.

func (*UploadDataSource) Info added in v1.8.0 

func (ud *UploadDataSource) Info() (ProcessingPhase, error)

Info is called to get initial information about the data.

func (*UploadDataSource) Process added in v1.8.0 

func (ud *UploadDataSource) Process() (ProcessingPhase, error)

Process is called to do any special processing before giving the url to the data back to the processor

func (*UploadDataSource) Transfer added in v1.8.0 

func (ud *UploadDataSource) Transfer(path string) (ProcessingPhase, error)

Transfer is called to transfer the data from the source to the passed in path.

func (*UploadDataSource) TransferFile added in v1.8.0 

func (ud *UploadDataSource) TransferFile(fileName string) (ProcessingPhase, error)

TransferFile is called to transfer the data from the source to the passed in file.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.