framework

package
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 26, 2024 License: Apache-2.0 Imports: 15 Imported by: 0

README

NodeResource Framework

Overview

The node resource controller is responsible for calculating the over-commit resources and updating the result on the node. The node resource framework defines some extension points for resource calculation and updating strategy. Each node resource plugin can implement one or more stages to reconcile the node resources.

The current extension stages provided by the node resource framework are as following:

  • Setup: It setups the plugin with options like controller client, scheme and event recorder.
type Plugin interface {
	Name() string
}

type SetupPlugin interface {
	Plugin
	Setup(opt *Option) error
}
  • Calculate: It calculates the node resources according to the Node and NodeMetric and generate a list of calculated node resource items. All node resource items will be merged into a NodeResource as the intermediate result of the Calculate stage. In case of the NodeMetric is abnormal, a plugin can implement the degraded calculation inside this stage. The Calculate plugin is also responsible for the reset of corresponding node resources when the colocation is configured as disabled.
type ResourceCalculatePlugin interface {
	Plugin
	Reset(node *Node, message string) []ResourceItem
	Calculate(strategy *ColocationStrategy, node *Node, podList *PodList, metrics *ResourceMetrics) ([]ResourceItem, error)
}

type ResourceItem struct {
	Name        ResourceName
	Quantity    *Quantity
	Labels      map[string]string
	Annotations map[string]string
	Message     string
	Reset       bool
}
  • PreUpdate: It allows the plugin to preprocess for the calculated results called before updating the Node. For example, a plugin can prepare and update some Objects like CRDs before updating the Node. And the plugin also can mutate the internal NodeResource object according the fully calculated results. It differs from the Prepare stage since a NodePreUpdatePlugin will be invoked only once in one loop (so the plugin should consider implement a retry login itself if needed), while the NodePreparePlugin is not expected to update other objects or mutate the NodeResource.
type NodePreUpdatePlugin interface {
	Plugin
	PreUpdate(strategy *ColocationStrategy, node *Node, nr *NodeResource) error
}
  • Prepare: It prepares the Node object with the calculated result NodeResource. Before the updating, it is invoked after the Calculate so to allow the plugin to retry when the client updates conflicts.
type NodePreparePlugin interface {
	Plugin
	Prepare(strategy *ColocationStrategy, node *Node, nr *NodeResource) error
}

type NodeResource struct {
	Resources   map[ResourceName]*Quantity
	Labels      map[string]string
	Annotations map[string]string
	Messages    map[ResourceName]string
	Resets      map[ResourceName]bool
}
  • NodeCheck: It checks if the newly-prepared Node object should be synchronized to the kube-apiserver. To be more specific, currently there are two types of NeedSync plugins for different client update methods, where one can determine whether the node status should be updated and another determines whether node metadata should be updated.
type NodeStatusCheckPlugin interface {
	Plugin
	NeedSync(strategy *ColocationStrategy, oldNode, newNode *Node) (bool, string)
}

type NodeMetaCheckPlugin interface {
	Plugin
	NeedSyncMeta(strategy *ColocationStrategy, oldNode, newNode *Node) (bool, string)
}

There is the workflow about how the node resource controller handles a dequeued Node object with plugins:

framework-img

Example: Batch Resource Plugin

The default BatchResource plugin is responsible for calculating and updating the Batch-tier resources. It implements the stages Setup, Calculate/Reset, PreUpdate, Prepare and NodeStatusCheck:

Setup:

In the initialization, the plugin sets the kube client, and add a watch for the NodeResourceTopology.

Calculate:

For each node, the plugin summarizes the resource allocated and the usage of high-priority (HP, priority classes higher than Batch) pods, then derives the allocatable resources of the Batch-tier with the formula:

batchAllocatable := nodeAllocatable * thresholdPercent - podUsage(HP) - systemUsage

Besides, the plugin implements the Reset method to clean up the Batch resources when the node colocation is disabled.

PreUpdate:

Before updating the Node obj, the plugin updates the zone-level Batch resources for the NRT (NodeResourceTopology) according to the calculated results from the Calculate stage.

Prepare:

The plugin sets the extended resources kubernetes.io/batch-cpu, kubernetes.io/batch-memory in the node.status.allocatable according to the calculated results from the Calculate/Reset stage.

NodeStatusCheck:

The plugin checks the extended resources kubernetes.io/batch-cpu, kubernetes.io/batch-memory of the prepared node and the old node. If the node's Batch resources have not been updated for too long or the calculated results changes too much, it will update the prepared Node object to the kube-apiserver.

What's More

The node resource framework is in Alpha. The defined stages may not be enough for some new scenarios. Please feel free to send Issues and PRs for improving the framework.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var AllPass = func(string) bool {
	return true
}

Functions

func RegisterNodeMetaCheckExtender added in v1.4.0 

func RegisterNodeMetaCheckExtender(filter FilterFn, plugins ...NodeMetaCheckPlugin)

func RegisterNodePreUpdateExtender added in v1.4.0 

func RegisterNodePreUpdateExtender(filter FilterFn, plugins ...NodePreUpdatePlugin)

func RegisterNodePrepareExtender 

func RegisterNodePrepareExtender(filter FilterFn, plugins ...NodePreparePlugin)

func RegisterNodeStatusCheckExtender added in v1.4.0 

func RegisterNodeStatusCheckExtender(filter FilterFn, plugins ...NodeStatusCheckPlugin)

func RegisterResourceCalculateExtender 

func RegisterResourceCalculateExtender(filter FilterFn, plugins ...ResourceCalculatePlugin)

func RegisterSetupExtender added in v1.3.0 

func RegisterSetupExtender(filter FilterFn, plugins ...SetupPlugin)

func RunNodeMetaCheckExtenders added in v1.4.0 

func RunNodeMetaCheckExtenders(strategy *configuration.ColocationStrategy, oldNode, newNode *corev1.Node) bool

func RunNodePreUpdateExtenders added in v1.4.0 

func RunNodePreUpdateExtenders(strategy *configuration.ColocationStrategy, node *corev1.Node, nr *NodeResource)

func RunNodePrepareExtenders 

func RunNodePrepareExtenders(strategy *configuration.ColocationStrategy, node *corev1.Node, nr *NodeResource)

func RunNodeStatusCheckExtenders added in v1.4.0 

func RunNodeStatusCheckExtenders(strategy *configuration.ColocationStrategy, oldNode, newNode *corev1.Node) bool

func RunResourceCalculateExtenders 

func RunResourceCalculateExtenders(nr *NodeResource, strategy *configuration.ColocationStrategy, node *corev1.Node,
	podList *corev1.PodList, resourceMetrics *ResourceMetrics)

func RunResourceResetExtenders 

func RunResourceResetExtenders(nr *NodeResource, node *corev1.Node, message string)

func RunSetupExtenders added in v1.3.0 

func RunSetupExtenders(opt *Option)

func UnregisterNodeMetaCheckExtender added in v1.4.0 

func UnregisterNodeMetaCheckExtender(name string)

func UnregisterNodePreUpdateExtender added in v1.4.0 

func UnregisterNodePreUpdateExtender(name string)

func UnregisterNodePrepareExtender 

func UnregisterNodePrepareExtender(name string)

func UnregisterNodeStatusCheckExtender added in v1.4.0 

func UnregisterNodeStatusCheckExtender(name string)

func UnregisterResourceCalculateExtender 

func UnregisterResourceCalculateExtender(name string)

func UnregisterSetupExtender added in v1.3.0 

func UnregisterSetupExtender(name string)

Types

type FilterFn added in v1.3.0 

type FilterFn func(string) bool

type NodeMetaCheckPlugin added in v1.4.0 

type NodeMetaCheckPlugin interface {
	Plugin
	NeedSyncMeta(strategy *configuration.ColocationStrategy, oldNode, newNode *corev1.Node) (bool, string)
}

type NodePreUpdatePlugin added in v1.4.0 

type NodePreUpdatePlugin interface {
	Plugin
	PreUpdate(strategy *configuration.ColocationStrategy, node *corev1.Node, nr *NodeResource) error
}

NodePreUpdatePlugin implements preprocessing for the calculated results called before updating the Node. There are mainly two use cases for this stage: 1. A plugin may prepare and update some Objects like CRDs before updating the Node obj (NodePrepare and NodeXXXCheck). 2. A plugin may need to mutate the internal NodeResource object before updating the Node object. It differs from the NodePreparePlugin in that a NodePreUpdatePlugin will be invoked only once in one loop (so the plugin should consider implement a retry login itself if needed), while the NodePreparePlugin is not expected to update other objects or mutate the NodeResource.

type NodePreparePlugin 

type NodePreparePlugin interface {
	Plugin
	Prepare(strategy *configuration.ColocationStrategy, node *corev1.Node, nr *NodeResource) error
}

NodePreparePlugin implements node resource preparing for the calculated results. For example, assign extended resources in the node allocatable. It is invoked each time the controller tries updating the latest NodeResource object with calculated results. NOTE: The Prepare should be idempotent since it can be called multiple times in one reconciliation.

type NodeResource 

type NodeResource struct {
	Resources     map[corev1.ResourceName]*resource.Quantity `json:"resources,omitempty"`
	ZoneResources map[string]corev1.ResourceList             `json:"zoneResources,omitempty"`
	Labels        map[string]string                          `json:"labels,omitempty"`
	Annotations   map[string]string                          `json:"annotations,omitempty"`
	Messages      map[corev1.ResourceName]string             `json:"messages,omitempty"`
	Resets        map[corev1.ResourceName]bool               `json:"resets,omitempty"`
}

func NewNodeResource 

func NewNodeResource(items ...ResourceItem) *NodeResource

func (*NodeResource) Delete 

func (nr *NodeResource) Delete(items ...ResourceItem)

func (*NodeResource) Get 

func (nr *NodeResource) Get(name corev1.ResourceName) *resource.Quantity

func (*NodeResource) Set 

func (nr *NodeResource) Set(items ...ResourceItem)

func (*NodeResource) SetResourceList 

func (nr *NodeResource) SetResourceList(rl corev1.ResourceList, message string)

type NodeStatusCheckPlugin added in v1.4.0 

type NodeStatusCheckPlugin interface {
	Plugin
	NeedSync(strategy *configuration.ColocationStrategy, oldNode, newNode *corev1.Node) (bool, string)
}

NodeStatusCheckPlugin implements the check of resource updating. For example, trigger an update if the values of the current is more than 10% different with the former.

type Option added in v1.3.0 

type Option struct {
	Client   client.Client
	Recorder record.EventRecorder
	Scheme   *runtime.Scheme
	Builder  *builder.Builder
}

func NewOption added in v1.3.0 

func NewOption() *Option

func (*Option) CompleteController added in v1.4.0 

func (o *Option) CompleteController(r reconcile.Reconciler) error

func (*Option) WithClient added in v1.3.0 

func (o *Option) WithClient(c client.Client) *Option

func (*Option) WithControllerBuilder added in v1.4.0 

func (o *Option) WithControllerBuilder(b *builder.Builder) *Option

func (*Option) WithManager added in v1.3.0 

func (o *Option) WithManager(mgr ctrl.Manager) *Option

func (*Option) WithRecorder added in v1.3.0 

func (o *Option) WithRecorder(r record.EventRecorder) *Option

func (*Option) WithScheme added in v1.3.0 

func (o *Option) WithScheme(s *runtime.Scheme) *Option

type Plugin 

type Plugin interface {
	Name() string
}

Plugin has its name. Plugins in a registry are executed in order of the registration.

type PluginConfig 

type PluginConfig struct {
	Plugin Plugin
}

type PluginRegistry 

type PluginRegistry struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func NewRegistry 

func NewRegistry(registryName string, plugins ...Plugin) *PluginRegistry

func (*PluginRegistry) GetAll 

func (r *PluginRegistry) GetAll() []Plugin

func (*PluginRegistry) MustRegister 

func (r *PluginRegistry) MustRegister(plugins ...Plugin)

func (*PluginRegistry) Size 

func (r *PluginRegistry) Size() int

func (*PluginRegistry) Unregister 

func (r *PluginRegistry) Unregister(name string)

type ResourceCalculatePlugin 

type ResourceCalculatePlugin interface {
	ResourceResetPlugin
	Calculate(strategy *configuration.ColocationStrategy, node *corev1.Node, podList *corev1.PodList, metrics *ResourceMetrics) ([]ResourceItem, error)
}

ResourceCalculatePlugin implements resource counting and overcommitment algorithms. It implements Reset which can be invoked when the calculated resources need a reset. A ResourceCalculatePlugin can handle the case when the metrics are abnormal by implementing degraded calculation.

type ResourceItem 

type ResourceItem struct {
	Name         corev1.ResourceName          `json:"name,omitempty"`
	Quantity     *resource.Quantity           `json:"quantity,omitempty"`
	ZoneQuantity map[string]resource.Quantity `json:"zoneQuantity,omitempty"`
	Labels       map[string]string            `json:"labels,omitempty"`
	Annotations  map[string]string            `json:"annotations,omitempty"`
	Message      string                       `json:"message,omitempty"` // the message about the resource calculation
	Reset        bool                         `json:"reset,omitempty"`   // whether to reset the resource or not
}

type ResourceMetrics 

type ResourceMetrics struct {
	NodeMetric *slov1alpha1.NodeMetric `json:"nodeMetric,omitempty"`
	// extended metrics
	Extensions *slov1alpha1.ExtensionsMap `json:"extensions,omitempty"`
}

type ResourceResetPlugin 

type ResourceResetPlugin interface {
	Plugin
	Reset(node *corev1.Node, message string) []ResourceItem
}

type SetupPlugin added in v1.3.0 

type SetupPlugin interface {
	Plugin
	Setup(opt *Option) error
}

SetupPlugin implements setup for the plugin. The framework exposes the kube ClientSet and controller builder to the plugins thus the plugins can set up their necessary clients, add new watches and initialize their internal states. The Setup of each plugin will be called before other extension stages and invoked only once.

type SyncContext 

type SyncContext struct {
	// contains filtered or unexported fields
}

func NewSyncContext 

func NewSyncContext() *SyncContext

func (*SyncContext) Delete 

func (s *SyncContext) Delete(key string)

func (*SyncContext) Load 

func (s *SyncContext) Load(key string) (time.Time, bool)

func (*SyncContext) Store 

func (s *SyncContext) Store(key string, value time.Time)

func (*SyncContext) WithContext 

func (s *SyncContext) WithContext(m map[string]time.Time) *SyncContext

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.