volumebinding

package
v1.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 9, 2024 License: Apache-2.0 Imports: 47 Imported by: 0

README

Introduction

When adapting to K8S V1.25, Volcano introduces the volumebinding package from K8S for self-maintenance. It is expected that the current package will be deleted from Volcano when adapting to K8S V1.29.

Background

Volcano encounters forward compatibility issues when adapting to K8S V1.25. Volcano can run properly in K8S V1.25, but cannot run in earlier K8S versions (for example, v1.23, v1.21). The error information is reflector.go:424] k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIStorageCapacity: the server could not find the requested resource.

Reason

The volumeBinder.csiStorageCapacityLister in the k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumebinding package uses the k8s.io/client-go/listers/storage/v1beta1 interface in V1.23, and is updated to the k8s.io/client-go/listers/storage/v1 interface in V1.25. (csiStorageCapacity is officially released v1 in v1.24) In addition, the EnableCSIStorageCapacity feature switch is removed.

The volumeBinder structure is statically defined. If the external plug-in imports the volumebinding package and uses the volumeBinder structure, forward compatibility issues may occur, for example, v1.CSIStorageCapacity is not defined.

Volcano references the volumebinding package of the K8S during scheduling for PVC binding. Therefore, after the K8S V1.25 is upgraded, the V1.23 and V1.21 versions cannot run properly.

Current solution

The k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumebinding package is introduced to Volcano for self-maintenance. The volumeBinder.csiStorageCapacityLister version is changed back to k8s.io/client-go/listers/storage/v1beta1 to implement compatibility processing of K8S v1.25, v1.23, and v1.21.

Future evolution

Solution 1: After the Volcano adapts to K8S V1.27, the volumeBinder.csiStorageCapacityLister interface version is upgraded from V1beta1 to V1, the self-maintained volumebinding package is deleted, and the volumebinding package in K8S is referenced again. After the upgrade, however, Only K8S V1.27 and V1.25 are compatible. Earlier versions, such as V1.23, are not compatible.

Solution 2: Add an abstract layer to volumeBinder in the volumebinding package and change the static version dependency to dynamic definition to ensure compatibility between v1beta1 and v1 interfaces.

Documentation

Index

Constants

View Source 
const (
	// ErrReasonBindConflict is used for VolumeBindingNoMatch predicate error.
	ErrReasonBindConflict ConflictReason = "node(s) didn't find available persistent volumes to bind"
	// ErrReasonNodeConflict is used for VolumeNodeAffinityConflict predicate error.
	ErrReasonNodeConflict ConflictReason = "node(s) had volume node affinity conflict"
	// ErrReasonNotEnoughSpace is used when a pod cannot start on a node because not enough storage space is available.
	ErrReasonNotEnoughSpace = "node(s) did not have enough free storage"
	// ErrReasonPVNotExist is used when a pod has one or more PVC(s) bound to non-existent persistent volume(s)"
	ErrReasonPVNotExist = "node(s) unavailable due to one or more pvc(s) bound to non-existent pv(s)"
)
View Source 
const Name = names.VolumeBinding

Name is the name of the plugin used in Registry and configurations.

Variables

This section is empty.

Functions

func New 

func New(plArgs runtime.Object, fh framework.Handle, fts feature.Features) (framework.Plugin, error)

New initializes a new plugin and returns it.

Types

type AssumeCache 

type AssumeCache interface {
	// Assume updates the object in-memory only
	Assume(obj interface{}) error

	// Restore the informer cache's version of the object
	Restore(objName string)

	// Get the object by name
	Get(objName string) (interface{}, error)

	// Get the API object by name
	GetAPIObj(objName string) (interface{}, error)

	// List all the objects in the cache
	List(indexObj interface{}) []interface{}
}

AssumeCache is a cache on top of the informer that allows for updating objects outside of informer events and also restoring the informer cache's version of the object. Objects are assumed to be Kubernetes API objects that implement meta.Interface

func NewAssumeCache 

func NewAssumeCache(informer cache.SharedIndexInformer, description, indexName string, indexFunc cache.IndexFunc) AssumeCache

NewAssumeCache creates an assume cache for general objects.

type BindingInfo 

type BindingInfo struct {
	// contains filtered or unexported fields
}

BindingInfo holds a binding between PV and PVC.

func (*BindingInfo) StorageClassName 

func (b *BindingInfo) StorageClassName() string

StorageClassName returns the name of the storage class.

func (*BindingInfo) StorageResource 

func (b *BindingInfo) StorageResource() *StorageResource

StorageResource returns storage resource.

type CapacityCheck 

type CapacityCheck struct {
	CSIDriverInformer          storageinformers.CSIDriverInformer
	CSIStorageCapacityInformer storageinformersv1beta1.CSIStorageCapacityInformer
}

CapacityCheck contains additional parameters for NewVolumeBinder that are only needed when checking volume sizes against available storage capacity is desired.

type ConflictReason 

type ConflictReason string

ConflictReason is used for the special strings which explain why volume binding is impossible for a node.

type ConflictReasons 

type ConflictReasons []ConflictReason

ConflictReasons contains all reasons that explain why volume binding is impossible for a node.

func (ConflictReasons) Len 

func (reasons ConflictReasons) Len() int

func (ConflictReasons) Less 

func (reasons ConflictReasons) Less(i, j int) bool

func (ConflictReasons) Swap 

func (reasons ConflictReasons) Swap(i, j int)

type FakeVolumeBinder 

type FakeVolumeBinder struct {
	AssumeCalled bool
	BindCalled   bool
	// contains filtered or unexported fields
}

FakeVolumeBinder represents a fake volume binder for testing.

func NewFakeVolumeBinder 

func NewFakeVolumeBinder(config *FakeVolumeBinderConfig) *FakeVolumeBinder

NewFakeVolumeBinder sets up all the caches needed for the scheduler to make topology-aware volume binding decisions.

func (*FakeVolumeBinder) AssumePodVolumes 

func (b *FakeVolumeBinder) AssumePodVolumes(assumedPod *v1.Pod, nodeName string, podVolumes *PodVolumes) (bool, error)

AssumePodVolumes implements SchedulerVolumeBinder.AssumePodVolumes.

func (*FakeVolumeBinder) BindPodVolumes 

func (b *FakeVolumeBinder) BindPodVolumes(ctx context.Context, assumedPod *v1.Pod, podVolumes *PodVolumes) error

BindPodVolumes implements SchedulerVolumeBinder.BindPodVolumes.

func (*FakeVolumeBinder) FindPodVolumes 

func (b *FakeVolumeBinder) FindPodVolumes(pod *v1.Pod, _ *PodVolumeClaims, node *v1.Node) (podVolumes *PodVolumes, reasons ConflictReasons, err error)

FindPodVolumes implements SchedulerVolumeBinder.FindPodVolumes.

func (*FakeVolumeBinder) GetEligibleNodes added in v1.8.1 

func (b *FakeVolumeBinder) GetEligibleNodes(boundClaims []*v1.PersistentVolumeClaim) (eligibleNodes sets.String)

GetEligibleNodes implements SchedulerVolumeBinder.GetEligibleNodes.

func (*FakeVolumeBinder) GetPodVolumeClaims added in v1.8.1 

func (b *FakeVolumeBinder) GetPodVolumeClaims(pod *v1.Pod) (podVolumeClaims *PodVolumeClaims, err error)

GetPodVolumeClaims implements SchedulerVolumeBinder.GetPodVolumes.

func (*FakeVolumeBinder) RevertAssumedPodVolumes 

func (b *FakeVolumeBinder) RevertAssumedPodVolumes(_ *PodVolumes)

RevertAssumedPodVolumes implements SchedulerVolumeBinder.RevertAssumedPodVolumes

type FakeVolumeBinderConfig 

type FakeVolumeBinderConfig struct {
	AllBound    bool
	FindReasons ConflictReasons
	FindErr     error
	AssumeErr   error
	BindErr     error
}

FakeVolumeBinderConfig holds configurations for fake volume binder.

type InTreeToCSITranslator 

type InTreeToCSITranslator interface {
	IsPVMigratable(pv *v1.PersistentVolume) bool
	GetInTreePluginNameFromSpec(pv *v1.PersistentVolume, vol *v1.Volume) (string, error)
	TranslateInTreePVToCSI(pv *v1.PersistentVolume) (*v1.PersistentVolume, error)
}

InTreeToCSITranslator contains methods required to check migratable status and perform translations from InTree PV's to CSI

type PVAssumeCache 

type PVAssumeCache interface {
	AssumeCache

	GetPV(pvName string) (*v1.PersistentVolume, error)
	GetAPIPV(pvName string) (*v1.PersistentVolume, error)
	ListPVs(storageClassName string) []*v1.PersistentVolume
}

PVAssumeCache is a AssumeCache for PersistentVolume objects

func NewPVAssumeCache 

func NewPVAssumeCache(informer cache.SharedIndexInformer) PVAssumeCache

NewPVAssumeCache creates a PV assume cache.

type PVCAssumeCache 

type PVCAssumeCache interface {
	AssumeCache

	// GetPVC returns the PVC from the cache with given pvcKey.
	// pvcKey is the result of MetaNamespaceKeyFunc on PVC obj
	GetPVC(pvcKey string) (*v1.PersistentVolumeClaim, error)
	GetAPIPVC(pvcKey string) (*v1.PersistentVolumeClaim, error)
}

PVCAssumeCache is a AssumeCache for PersistentVolumeClaim objects

func NewPVCAssumeCache 

func NewPVCAssumeCache(informer cache.SharedIndexInformer) PVCAssumeCache

NewPVCAssumeCache creates a PVC assume cache.

type PodVolumeClaims added in v1.8.1 

type PodVolumeClaims struct {
	// contains filtered or unexported fields
}

type PodVolumes 

type PodVolumes struct {
	// StaticBindings are binding decisions for PVCs which can be bound to
	// pre-provisioned static PVs.
	StaticBindings []*BindingInfo
	// DynamicProvisions are PVCs that require dynamic provisioning
	DynamicProvisions []*v1.PersistentVolumeClaim
}

PodVolumes holds pod's volumes information used in volume scheduling.

type SchedulerVolumeBinder 

type SchedulerVolumeBinder interface {
	// GetPodVolumeClaims returns a pod's PVCs separated into bound, unbound with delayed binding (including provisioning),
	// unbound with immediate binding (including prebound) and PVs that belong to storage classes of unbound PVCs with delayed binding.
	GetPodVolumeClaims(pod *v1.Pod) (podVolumeClaims *PodVolumeClaims, err error)

	// GetEligibleNodes checks the existing bound claims of the pod to determine if the list of nodes can be
	// potentially reduced down to a subset of eligible nodes based on the bound claims which then can be used
	// in subsequent scheduling stages.
	//
	// If eligibleNodes is 'nil', then it indicates that such eligible node reduction cannot be made
	// and all nodes should be considered.
	GetEligibleNodes(boundClaims []*v1.PersistentVolumeClaim) (eligibleNodes sets.String)

	// FindPodVolumes checks if all of a Pod's PVCs can be satisfied by the
	// node and returns pod's volumes information.
	//
	// If a PVC is bound, it checks if the PV's NodeAffinity matches the Node.
	// Otherwise, it tries to find an available PV to bind to the PVC.
	//
	// It returns an error when something went wrong or a list of reasons why the node is
	// (currently) not usable for the pod.
	//
	// If the CSIStorageCapacity feature is enabled, then it also checks for sufficient storage
	// for volumes that still need to be created.
	//
	// This function is called by the scheduler VolumeBinding plugin and can be called in parallel
	FindPodVolumes(pod *v1.Pod, podVolumeClaims *PodVolumeClaims, node *v1.Node) (podVolumes *PodVolumes, reasons ConflictReasons, err error)

	// AssumePodVolumes will:
	// 1. Take the PV matches for unbound PVCs and update the PV cache assuming
	// that the PV is prebound to the PVC.
	// 2. Take the PVCs that need provisioning and update the PVC cache with related
	// annotations set.
	//
	// It returns true if all volumes are fully bound
	//
	// This function is called serially.
	AssumePodVolumes(assumedPod *v1.Pod, nodeName string, podVolumes *PodVolumes) (allFullyBound bool, err error)

	// RevertAssumedPodVolumes will revert assumed PV and PVC cache.
	RevertAssumedPodVolumes(podVolumes *PodVolumes)

	// BindPodVolumes will:
	// 1. Initiate the volume binding by making the API call to prebind the PV
	// to its matching PVC.
	// 2. Trigger the volume provisioning by making the API call to set related
	// annotations on the PVC
	// 3. Wait for PVCs to be completely bound by the PV controller
	//
	// This function can be called in parallel.
	BindPodVolumes(ctx context.Context, assumedPod *v1.Pod, podVolumes *PodVolumes) error
}

SchedulerVolumeBinder is used by the scheduler VolumeBinding plugin to handle PVC/PV binding and dynamic provisioning. The binding decisions are integrated into the pod scheduling workflow so that the PV NodeAffinity is also considered along with the pod's other scheduling requirements.

This integrates into the existing scheduler workflow as follows:

  1. The scheduler takes a Pod off the scheduler queue and processes it serially: a. Invokes all pre-filter plugins for the pod. GetPodVolumes() is invoked here, pod volume information will be saved in current scheduling cycle state for later use. If pod has bound immediate PVCs, GetEligibleNodes() is invoked to potentially reduce down the list of eligible nodes based on the bound PV's NodeAffinity (if any). b. Invokes all filter plugins, parallelized across nodes. FindPodVolumes() is invoked here. c. Invokes all score plugins. Future/TBD d. Selects the best node for the Pod. e. Invokes all reserve plugins. AssumePodVolumes() is invoked here. i. If PVC binding is required, cache in-memory only: * For manual binding: update PV objects for prebinding to the corresponding PVCs. * For dynamic provisioning: update PVC object with a selected node from c) * For the pod, which PVCs and PVs need API updates. ii. Afterwards, the main scheduler caches the Pod->Node binding in the scheduler's pod cache, This is handled in the scheduler and not here. f. Asynchronously bind volumes and pod in a separate goroutine i. BindPodVolumes() is called first in PreBind phase. It makes all the necessary API updates and waits for PV controller to fully bind and provision the PVCs. If binding fails, the Pod is sent back through the scheduler. ii. After BindPodVolumes() is complete, then the scheduler does the final Pod->Node binding.
  2. Once all the assume operations are done in e), the scheduler processes the next Pod in the scheduler queue while the actual binding operation occurs in the background.

func NewVolumeBinder 

func NewVolumeBinder(
	kubeClient clientset.Interface,
	podInformer coreinformers.PodInformer,
	nodeInformer coreinformers.NodeInformer,
	csiNodeInformer storageinformers.CSINodeInformer,
	pvcInformer coreinformers.PersistentVolumeClaimInformer,
	pvInformer coreinformers.PersistentVolumeInformer,
	storageClassInformer storageinformers.StorageClassInformer,
	capacityCheck *CapacityCheck,
	bindTimeout time.Duration) SchedulerVolumeBinder

NewVolumeBinder sets up all the caches needed for the scheduler to make volume binding decisions.

capacityCheck determines how storage capacity is checked (CSIStorageCapacity feature).

type StorageResource 

type StorageResource struct {
	Requested int64
	Capacity  int64
}

StorageResource represents storage resource.

type VolumeBinding 

type VolumeBinding struct {
	Binder    SchedulerVolumeBinder
	PVCLister corelisters.PersistentVolumeClaimLister
	// contains filtered or unexported fields
}

VolumeBinding is a plugin that binds pod volumes in scheduling. In the Filter phase, pod binding cache is created for the pod and used in Reserve and PreBind phases.

func (*VolumeBinding) EventsToRegister 

func (pl *VolumeBinding) EventsToRegister() []framework.ClusterEvent

EventsToRegister returns the possible events that may make a Pod failed by this plugin schedulable.

func (*VolumeBinding) Filter 

func (pl *VolumeBinding) Filter(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeInfo *framework.NodeInfo) *framework.Status

Filter invoked at the filter extension point. It evaluates if a pod can fit due to the volumes it requests, for both bound and unbound PVCs.

For PVCs that are bound, then it checks that the corresponding PV's node affinity is satisfied by the given node.

For PVCs that are unbound, it tries to find available PVs that can satisfy the PVC requirements and that the PV node affinity is satisfied by the given node.

If storage capacity tracking is enabled, then enough space has to be available for the node and volumes that still need to be created.

The predicate returns true if all bound PVCs have compatible PVs with the node, and if all unbound PVCs can be matched with an available and node-compatible PV.

func (*VolumeBinding) Name 

func (pl *VolumeBinding) Name() string

Name returns name of the plugin. It is used in logs, etc.

func (*VolumeBinding) PreBind 

func (pl *VolumeBinding) PreBind(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeName string) *framework.Status

PreBind will make the API update with the assumed bindings and wait until the PV controller has completely finished the binding operation.

If binding errors, times out or gets undone, then an error will be returned to retry scheduling.

func (*VolumeBinding) PreFilter 

func (pl *VolumeBinding) PreFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status)

PreFilter invoked at the prefilter extension point to check if pod has all immediate PVCs bound. If not all immediate PVCs are bound, an UnschedulableAndUnresolvable is returned.

func (*VolumeBinding) PreFilterExtensions 

func (pl *VolumeBinding) PreFilterExtensions() framework.PreFilterExtensions

PreFilterExtensions returns prefilter extensions, pod add and remove.

func (*VolumeBinding) Reserve 

func (pl *VolumeBinding) Reserve(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeName string) *framework.Status

Reserve reserves volumes of pod and saves binding status in cycle state.

func (*VolumeBinding) Score 

func (pl *VolumeBinding) Score(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status)

Score invoked at the score extension point.

func (*VolumeBinding) ScoreExtensions 

func (pl *VolumeBinding) ScoreExtensions() framework.ScoreExtensions

ScoreExtensions of the Score plugin.

func (*VolumeBinding) Unreserve 

func (pl *VolumeBinding) Unreserve(ctx context.Context, cs *framework.CycleState, pod *v1.Pod, nodeName string)

Unreserve clears assumed PV and PVC cache. It's idempotent, and does nothing if no cache found for the given pod.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.