base

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 18, 2024 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckReferenceDataset added in v0.9.0 

func CheckReferenceDataset(dataset *datav1alpha1.Dataset) (check bool, err error)

func GetDataOperationKey added in v0.9.0 

func GetDataOperationKey(object client.Object) string

func GetDatasetRefName added in v0.9.0 

func GetDatasetRefName(name, namespace string) string

func GetMetadataListFromAnnotation added in v0.9.0 

func GetMetadataListFromAnnotation(accessor metav1.ObjectMetaAccessor) (ret []datav1alpha1.Metadata)

func GetPhysicalDatasetFromMounts added in v1.0.0 

func GetPhysicalDatasetFromMounts(mounts []datav1alpha1.Mount) []types.NamespacedName

func GetPhysicalDatasetSubPath added in v1.0.0 

func GetPhysicalDatasetSubPath(virtualDataset *datav1alpha1.Dataset) []string

func GetRuntimeStatus added in v0.9.0 

func GetRuntimeStatus(client client.Client, runtimeType, name, namespace string) (status *datav1alpha1.RuntimeStatus, err error)

func InstallDataOperationHelmIfNotExist added in v0.9.0 

func InstallDataOperationHelmIfNotExist(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface,
	yamlGenerator DataOperatorYamlGenerator) (err error)

func RecordDatasetMetrics added in v0.9.0 

func RecordDatasetMetrics(result MetadataSyncResult, datasetNamespace, datasetName string, log logr.Logger)

RecordDatasetMetrics records dataset-related metrics from the given MetadataSyncResult

func ReleaseTargetDataset added in v0.9.0 

func ReleaseTargetDataset(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface) error

ReleaseTargetDataset release target dataset OperationRef field which marks the data operation being performed.

func SafeClose added in v0.9.0 

func SafeClose(ch chan MetadataSyncResult) (closed bool)

SafeClose closes the metadataSyncResultChannel but ignores panic when the channel is already closed. Returns true if the channel is already closed.

func SafeSend added in v0.9.0 

func SafeSend(ch chan MetadataSyncResult, result MetadataSyncResult) (closed bool)

SafeSend sends result to the metadataSyncResultChannel but ignores panic when the channel is already closed Returns true if the channel is already closed.

func SetDataOperationInTargetDataset added in v0.9.0 

func SetDataOperationInTargetDataset(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface, engine Engine) error

SetDataOperationInTargetDataset set status of target dataset to mark the data operation being performed.

Types

type CachePath added in v0.5.0 

type CachePath struct {
	Path string

	Quota *resource.Quantity
}

type DataOperator added in v0.9.0 

type DataOperator interface {
	Operate(ctx cruntime.ReconcileRequestContext, opStatus *datav1alpha1.OperationStatus, operation dataoperation.OperationInterface) (ctrl.Result, error)
}

DataOperator is a common interface of TemplateEngine for Data Operations like DataBackup/DataLoad/DataMigrate etc.

type DataOperatorYamlGenerator added in v0.9.0 

type DataOperatorYamlGenerator interface {
	GetDataOperationValueFile(ctx cruntime.ReconcileRequestContext, operation dataoperation.OperationInterface) (valueFileName string, err error)
}

DataOperatorYamlGenerator is the implementation of DataOperator interface for runtime engine

type Databackuper added in v0.9.0 

type Databackuper interface {
	BackupData(ctx cruntime.ReconcileRequestContext, targetDataBackup datav1alpha1.DataBackup) (ctrl.Result, error)
}

type Dataloader added in v0.6.0 

type Dataloader interface {
	// LoadData generate dataload values and install helm chart
	LoadData(ctx cruntime.ReconcileRequestContext, targetDataload datav1alpha1.DataLoad) (err error)

	// CheckRuntimeReady Check if runtime is ready
	// @Deprecated because it's common for all engine
	CheckRuntimeReady() (ready bool)
}

type Datamigrater added in v0.9.0 

type Datamigrater interface {
	// MigrateData generate datamigrate values and install helm chart
	MigrateData(ctx cruntime.ReconcileRequestContext, targetDataMigrate datav1alpha1.DataMigrate) (err error)
}

type Engine 

type Engine interface {
	// ID returns the id
	ID() string

	// Shutdown and clean up the engine
	Shutdown() error

	// Setup the engine
	Setup(ctx cruntime.ReconcileRequestContext) (ready bool, err error)

	// Setup the Volume
	CreateVolume() (err error)

	// Destroy the Volume
	DeleteVolume() (err error)

	// Sync syncs the alluxio runtime
	Sync(ctx cruntime.ReconcileRequestContext) error

	// Dataloader
	// @Deprecated use DataOperator instead.
	Dataloader

	// Datamigrater
	// @Deprecated use DataOperator instead.
	Datamigrater

	// DataOperator is a common interface for Data Operations like DataBackup/DataLoad/DataMigrate etc.
	DataOperator
}

Engine interface defines the interfaces that should be implemented by a distributed data caching Engine. Thread safety is required from implementations of this interface.

type Fuse added in v0.5.0 

type Fuse struct {
	// fuse is deployed in global mode
	Global bool

	NodeSelector map[string]string

	// CleanPolicy decides when to clean fuse pods.
	CleanPolicy datav1alpha1.FuseCleanPolicy
}

type Implement 

type Implement interface {
	UnderFileSystemService

	DataOperatorYamlGenerator

	// ShouldSetupMaster checks if the master ready
	CheckMasterReady() (ready bool, err error)

	// CheckWorkersReady checks if the workers ready
	CheckWorkersReady() (ready bool, err error)

	// ShouldSetupMaster checks if we need to setup the master
	ShouldSetupMaster() (should bool, err error)

	// ShouldSetupWorkers checks if we need to setup the workers
	ShouldSetupWorkers() (should bool, err error)

	// ShouldCheckUFS checks if we should check the ufs
	ShouldCheckUFS() (should bool, err error)

	// SetupMaster setup the cache master
	SetupMaster() (err error)

	// SetupWorkers setup the cache worker
	SetupWorkers() (err error)

	// UpdateDatasetStatus update the status of Dataset according to the given phase
	UpdateDatasetStatus(phase datav1alpha1.DatasetPhase) (err error)

	// PrepareUFS prepare the mounts and metadata if it's not ready
	PrepareUFS() (err error)

	// ShouldUpdateUFS check if we need to update the ufs and return all ufs to update
	// If the ufs have changed and the engine supports add/remove mount points dynamically,
	// then we need to UpdateOnUFSChange
	ShouldUpdateUFS() (ufsToUpdate *utils.UFSToUpdate)

	// UpdateOnUFSChange update the mount point of Dataset if ufs change
	// if an engine doesn't support UpdateOnUFSChange, it need to return false
	UpdateOnUFSChange(ufsToUpdate *utils.UFSToUpdate) (ready bool, err error)

	// Shutdown and clean up the engine
	Shutdown() error

	// AssignNodesToCache picks up the nodes for replicas
	AssignNodesToCache(desiredNum int32) (currentNum int32, err error)

	// CheckRuntimeHealthy checks runtime healthy
	CheckRuntimeHealthy() (err error)

	// UpdateCacheOfDataset updates cache of the dataset
	UpdateCacheOfDataset() (err error)

	// CheckAndUpdateRuntimeStatus checks and updates the status
	CheckAndUpdateRuntimeStatus() (ready bool, err error)

	// CreateVolume create the pv and pvc for the Dataset
	CreateVolume() error

	// SyncReplicas syncs the replicas
	SyncReplicas(ctx cruntime.ReconcileRequestContext) error

	// SyncMetadata syncs all metadata from UFS
	SyncMetadata() (err error)

	// DeleteVolume Destroy the Volume
	DeleteVolume() (err error)

	// BindToDataset binds the engine to dataset
	BindToDataset() (err error)

	// CreateDataLoadJob creates the job to load data
	// @Deprecated TODO: remove when DataOperator ready
	CreateDataLoadJob(ctx cruntime.ReconcileRequestContext, targetDataload datav1alpha1.DataLoad) error

	// CreateDataMigrateJob creates the job to load data
	// @Deprecated TODO: remove when DataOperator ready
	CreateDataMigrateJob(ctx cruntime.ReconcileRequestContext, targetDataMigrate datav1alpha1.DataMigrate) error

	// checks if the runtime is ready
	CheckRuntimeReady() (ready bool)

	// SyncRuntime syncs the runtime spec
	SyncRuntime(ctx cruntime.ReconcileRequestContext) (changed bool, err error)

	// Sync the scheduleInfo to cacheNodes
	SyncScheduleInfoToCacheNodes() (err error)
}

Implement is what the real engine should implement if it use the TemplateEngine

type Level added in v0.5.0 

type Level struct {
	MediumType common.MediumType

	VolumeType common.VolumeType

	VolumeSource datav1alpha1.VolumeSource

	CachePaths []CachePath

	High string

	Low string
}

type MetadataSyncResult added in v0.9.0 

type MetadataSyncResult struct {
	Done      bool
	StartTime time.Time
	UfsTotal  string
	FileNum   string
	Err       error
}

MetadataSyncResult describes result for asynchronous metadata sync

type RuntimeInfo added in v0.5.0 

type RuntimeInfo struct {
	// contains filtered or unexported fields
}

The real Runtime Info should implement

func (*RuntimeInfo) GetCommonLabelName added in v0.6.0 

func (info *RuntimeInfo) GetCommonLabelName() string

func (*RuntimeInfo) GetDatasetNumLabelName added in v0.6.0 

func (info *RuntimeInfo) GetDatasetNumLabelName() string

GetDatasetNumLabelname get the label to record how much datasets on a node

func (*RuntimeInfo) GetFuseCleanPolicy added in v0.7.0 

func (info *RuntimeInfo) GetFuseCleanPolicy() datav1alpha1.FuseCleanPolicy

func (*RuntimeInfo) GetFuseContainerTemplate added in v1.0.0 

func (info *RuntimeInfo) GetFuseContainerTemplate() (template *common.FuseInjectionTemplate, err error)

GetFuseContainerTemplate collects the fuse container spec from the runtime's fuse daemonSet spec. The function summarizes fuse related information into the template and returns it. The template then can be freely modified according to need of the serverless platform.

func (*RuntimeInfo) GetFuseDeployMode added in v0.5.0 

func (info *RuntimeInfo) GetFuseDeployMode() (global bool, nodeSelector map[string]string)

GetFuseDeployMode gets the fuse deploy mode

func (*RuntimeInfo) GetFuseLabelName added in v0.7.0 

func (info *RuntimeInfo) GetFuseLabelName() string

GetFuseLabelName gets the label indicating a fuse running on some node.

func (*RuntimeInfo) GetLabelNameForDisk added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForDisk() string

func (*RuntimeInfo) GetLabelNameForMemory added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForMemory() string

func (*RuntimeInfo) GetLabelNameForTotal added in v0.6.0 

func (info *RuntimeInfo) GetLabelNameForTotal() string

func (*RuntimeInfo) GetMetadataList added in v0.9.0 

func (info *RuntimeInfo) GetMetadataList() []datav1alpha1.Metadata

func (*RuntimeInfo) GetName added in v0.5.0 

func (info *RuntimeInfo) GetName() string

GetName gets name

func (*RuntimeInfo) GetNamespace added in v0.5.0 

func (info *RuntimeInfo) GetNamespace() string

GetNamespace gets namespace

func (*RuntimeInfo) GetPersistentVolumeName added in v0.6.0 

func (info *RuntimeInfo) GetPersistentVolumeName() string

func (*RuntimeInfo) GetRuntimeLabelName added in v0.6.0 

func (info *RuntimeInfo) GetRuntimeLabelName() string

func (*RuntimeInfo) GetRuntimeType added in v0.5.0 

func (info *RuntimeInfo) GetRuntimeType() string

GetRuntimeType gets runtime type

func (*RuntimeInfo) GetTieredStoreInfo added in v0.6.0 

func (info *RuntimeInfo) GetTieredStoreInfo() TieredStoreInfo

func (*RuntimeInfo) IsDeprecatedNodeLabel added in v0.6.0 

func (info *RuntimeInfo) IsDeprecatedNodeLabel() bool

IsDeprecatedNodeLabel checks if using deprecated node label

func (*RuntimeInfo) IsDeprecatedPVName added in v0.6.0 

func (info *RuntimeInfo) IsDeprecatedPVName() bool

func (*RuntimeInfo) IsExclusive added in v0.5.0 

func (info *RuntimeInfo) IsExclusive() bool

IsExclusive determines if the runtime is exlusive

func (*RuntimeInfo) SetClient added in v0.7.0 

func (info *RuntimeInfo) SetClient(client client.Client)

func (*RuntimeInfo) SetDeprecatedNodeLabel added in v0.6.0 

func (info *RuntimeInfo) SetDeprecatedNodeLabel(deprecated bool)

SetDeprecatedNodeLabel set the DeprecatedNodeLabel

func (*RuntimeInfo) SetDeprecatedPVName added in v0.6.0 

func (info *RuntimeInfo) SetDeprecatedPVName(deprecated bool)

func (*RuntimeInfo) SetupFuseCleanPolicy added in v0.7.0 

func (info *RuntimeInfo) SetupFuseCleanPolicy(policy datav1alpha1.FuseCleanPolicy)

func (*RuntimeInfo) SetupFuseDeployMode added in v0.5.0 

func (info *RuntimeInfo) SetupFuseDeployMode(global bool, nodeSelector map[string]string)

SetupFuseDeployMode setups the fuse deploy mode

func (*RuntimeInfo) SetupWithDataset added in v0.5.0 

func (info *RuntimeInfo) SetupWithDataset(dataset *datav1alpha1.Dataset)

SetupWithDataset determines if need to setup with the info of dataset

type RuntimeInfoInterface added in v0.5.0 

type RuntimeInfoInterface interface {
	GetTieredStoreInfo() TieredStoreInfo

	GetName() string

	GetNamespace() string

	GetRuntimeType() string

	GetLabelNameForMemory() string

	GetLabelNameForDisk() string

	GetLabelNameForTotal() string

	GetCommonLabelName() string

	GetFuseLabelName() string

	GetRuntimeLabelName() string

	GetDatasetNumLabelName() string

	GetPersistentVolumeName() string

	IsExclusive() bool

	SetupFuseDeployMode(global bool, nodeSelector map[string]string)

	SetupFuseCleanPolicy(policy datav1alpha1.FuseCleanPolicy)

	SetupWithDataset(dataset *datav1alpha1.Dataset)

	GetFuseDeployMode() (global bool, nodeSelector map[string]string)

	GetFuseCleanPolicy() datav1alpha1.FuseCleanPolicy

	SetDeprecatedNodeLabel(deprecated bool)

	IsDeprecatedNodeLabel() bool

	SetDeprecatedPVName(deprecated bool)

	IsDeprecatedPVName() bool

	GetFuseContainerTemplate() (template *common.FuseInjectionTemplate, err error)

	SetClient(client client.Client)

	GetMetadataList() []datav1alpha1.Metadata
}

Runtime Information interface defines the interfaces that should be implemented by Alluxio Runtime or other implementation . Thread safety is required from implementations of this interface.

func BuildRuntimeInfo added in v0.5.0 

func BuildRuntimeInfo(name string,
	namespace string,
	runtimeType string,
	tieredstore datav1alpha1.TieredStore,
	opts ...RuntimeInfoOption) (runtime RuntimeInfoInterface, err error)

func GetRuntimeInfo added in v0.6.0 

func GetRuntimeInfo(client client.Client, name, namespace string) (runtimeInfo RuntimeInfoInterface, err error)

GetRuntimeInfo gets the RuntimeInfo according to name and namespace of it

type RuntimeInfoOption added in v0.9.0 

type RuntimeInfoOption func(info *RuntimeInfo)

func WithMetadataList added in v0.9.0 

func WithMetadataList(metadataList []datav1alpha1.Metadata) RuntimeInfoOption

type RuntimeInterface added in v0.7.0 

type RuntimeInterface interface {

	// Replicas gets the replicas of runtime worker
	Replicas() int32

	// GetStatus gets the status of runtime
	GetStatus() *datav1alpha1.RuntimeStatus

	client.Object
}

type TemplateEngine 

type TemplateEngine struct {
	Implement
	Id string
	client.Client
	Log     logr.Logger
	Context cruntime.ReconcileRequestContext
	// contains filtered or unexported fields
}

func NewTemplateEngine 

func NewTemplateEngine(impl Implement,
	id string,

	context cruntime.ReconcileRequestContext) *TemplateEngine

NewTemplateEngine creates template engine

func (*TemplateEngine) CheckRuntimeReady added in v0.6.0 

func (t *TemplateEngine) CheckRuntimeReady() (ready bool)

func (*TemplateEngine) CreateVolume 

func (t *TemplateEngine) CreateVolume() (err error)

Setup the CSI

func (*TemplateEngine) DeleteVolume 

func (t *TemplateEngine) DeleteVolume() (err error)

Setup the CSI

func (*TemplateEngine) ID 

func (t *TemplateEngine) ID() string

ID returns the id of the engine

func (*TemplateEngine) LoadData added in v0.6.0 

func (t *TemplateEngine) LoadData(ctx cruntime.ReconcileRequestContext, targetDataload datav1alpha1.DataLoad) (err error)

func (*TemplateEngine) MigrateData added in v0.9.0 

func (t *TemplateEngine) MigrateData(ctx cruntime.ReconcileRequestContext, targetDataMigrate datav1alpha1.DataMigrate) (err error)

func (*TemplateEngine) Operate added in v0.9.0 

func (t *TemplateEngine) Operate(ctx cruntime.ReconcileRequestContext, opStatus *datav1alpha1.OperationStatus,
	operation dataoperation.OperationInterface) (ctrl.Result, error)

func (*TemplateEngine) Setup 

func (b *TemplateEngine) Setup(ctx cruntime.ReconcileRequestContext) (ready bool, err error)

Setup the ddc engine

func (*TemplateEngine) Shutdown 

func (t *TemplateEngine) Shutdown() error

Shutdown and clean up the engine

func (*TemplateEngine) Sync 

func (t *TemplateEngine) Sync(ctx cruntime.ReconcileRequestContext) (err error)

SyncReplicas syncs the replicas

type TieredStoreInfo added in v0.6.0 

type TieredStoreInfo struct {
	Levels []Level
}

type UnderFileSystemService 

type UnderFileSystemService interface {
	UsedStorageBytes() (int64, error)

	FreeStorageBytes() (int64, error)

	TotalStorageBytes() (int64, error)

	TotalFileNums() (int64, error)
}

UnderFileSystemService interface defines the interfaces that should be implemented by a underlayer fileSystem service for the data. The implementation is the underlayer file system connector. It is responsible for checking ufs and preload the data. Thread safety is required from implementations of this interface.

Source Files

View all Source files

Directories

Path Synopsis
Package base is a generated GoMock package.
 Package base is a generated GoMock package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.