objects

package
v1.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 8, 2024 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	Soft string = "Soft"
	Hard string = "Hard"

	NotEnoughUserQuota  = "Not enough user quota"
	NotEnoughQueueQuota = "Not enough queue quota"
)
View Source 
const (
	RunApplication applicationEvent = iota
	RejectApplication
	CompleteApplication
	FailApplication
	ExpireApplication
	ResumeApplication
)
View Source 
const (
	New applicationState = iota
	Accepted
	Starting
	Running
	Rejected
	Completing
	Completed
	Failing
	Failed
	Expired
	Resuming
)

Application states are used for filtering in the webservice handlers. Please check&update the logic as needed if the state machine is modified

View Source 
const (
	EventNotNeeded = -1
)
View Source 
const (
	UnknownInstanceType = "UNKNOWN"
)

Variables

This section is empty.

Functions

func CloneAllocationTags added in v1.1.0 

func CloneAllocationTags(tags map[string]string) map[string]string

CloneAllocationTags clones a tag map for safe copying

func NewAppState 

func NewAppState() *fsm.FSM

func NewObjectState 

func NewObjectState() *fsm.FSM

func NewTreeIterator added in v1.4.0 

func NewTreeIterator(accept func(*Node) bool, getTree func() *btree.BTree) *treeIterator

func SetCompletingTimeout 

func SetCompletingTimeout(duration time.Duration)

SetCompletingTimeout should be used for testing only.

func SetReservationDelay 

func SetReservationDelay(delay time.Duration)

Set the reservation delay. Set when the cluster context is created to disable reservation.

Types

type Allocation 

type Allocation struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewAllocation 

func NewAllocation(nodeID string, ask *AllocationAsk) *Allocation

func NewAllocationFromSI 

func NewAllocationFromSI(alloc *si.Allocation) *Allocation

Create a new Allocation from a node recovered allocation. Also creates an AllocationAsk to maintain backward compatible behaviour This returns a nil Allocation on nil input or errors

func (*Allocation) AddRelease added in v1.1.0 

func (a *Allocation) AddRelease(release *Allocation)

AddRelease adds a new release to this allocation

func (*Allocation) ClearReleases added in v1.1.0 

func (a *Allocation) ClearReleases()

ClearReleases removes all releases from this allocation

func (*Allocation) GetAllocatedResource added in v1.1.0 

func (a *Allocation) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a reference to the allocated resources for this allocation. This must be treated as read-only.

func (*Allocation) GetAllocationID added in v1.5.0 

func (a *Allocation) GetAllocationID() string

GetAllocationID returns the allocationID for this allocation

func (*Allocation) GetAllocationKey added in v1.1.0 

func (a *Allocation) GetAllocationKey() string

GetAllocationKey returns the allocation key of this allocation

func (*Allocation) GetApplicationID added in v1.1.0 

func (a *Allocation) GetApplicationID() string

GetApplicationID returns the application ID for this allocation

func (*Allocation) GetAsk added in v1.1.0 

func (a *Allocation) GetAsk() *AllocationAsk

GetAsk returns the ask associated with this allocation

func (*Allocation) GetBindTime added in v1.3.0 

func (a *Allocation) GetBindTime() time.Time

GetBindTime returns the time this allocation was created

func (*Allocation) GetCreateTime 

func (a *Allocation) GetCreateTime() time.Time

GetCreateTime returns the time this allocation was created

func (*Allocation) GetFirstRelease added in v1.1.0 

func (a *Allocation) GetFirstRelease() *Allocation

GetFirstRelease returns the first release for this allocation

func (*Allocation) GetInstanceType added in v1.3.0 

func (a *Allocation) GetInstanceType() string

GetInstanceType return the type of the instance used by this allocation

func (*Allocation) GetNodeID added in v1.1.0 

func (a *Allocation) GetNodeID() string

GetNodeID gets the node this allocation is assigned to

func (*Allocation) GetPartitionName added in v1.1.0 

func (a *Allocation) GetPartitionName() string

GetPartitionName returns the partition name for this allocation

func (*Allocation) GetPlaceholderCreateTime 

func (a *Allocation) GetPlaceholderCreateTime() time.Time

GetPlaceholderCreateTime returns the placeholder's create time for this alloc, if applicable

func (*Allocation) GetPriority added in v1.1.0 

func (a *Allocation) GetPriority() int32

GetPriority returns the priority of this allocation

func (*Allocation) GetReleaseCount added in v1.1.0 

func (a *Allocation) GetReleaseCount() int

GetReleaseCount gets the number of releases associated with this allocation

func (*Allocation) GetReleasesClone added in v1.1.0 

func (a *Allocation) GetReleasesClone() []*Allocation

GetReleasesClone returns a clone of the release list

func (*Allocation) GetReservedNodeID added in v1.1.0 

func (a *Allocation) GetReservedNodeID() string

GetReservedNodeID gets the node this allocation is reserved for

func (*Allocation) GetResult added in v1.1.0 

func (a *Allocation) GetResult() AllocationResult

GetResult gets the result of this allocation

func (*Allocation) GetTagsClone added in v1.1.0 

func (a *Allocation) GetTagsClone() map[string]string

GetTagsClone returns the copy of the tags for this allocation

func (*Allocation) GetTaskGroup added in v1.1.0 

func (a *Allocation) GetTaskGroup() string

GetTaskGroup returns the task group name for this allocation

func (*Allocation) IsPlaceholder 

func (a *Allocation) IsPlaceholder() bool

IsPlaceholder returns whether the allocation is a placeholder

func (*Allocation) IsPlaceholderUsed added in v1.1.0 

func (a *Allocation) IsPlaceholderUsed() bool

IsPlaceholderUsed returns whether this alloc is replacing a placeholder

func (*Allocation) IsPreempted added in v1.2.0 

func (a *Allocation) IsPreempted() bool

IsPreempted returns whether the allocation has been marked for preemption or not.

func (*Allocation) IsReleased 

func (a *Allocation) IsReleased() bool

IsReleased returns the release status of the allocation

func (*Allocation) MarkPreempted added in v1.2.0 

func (a *Allocation) MarkPreempted()

MarkPreempted marks the allocation as preempted.

func (*Allocation) NewSIFromAllocation 

func (a *Allocation) NewSIFromAllocation() *si.Allocation

Convert the Allocation into a SI object. This is a limited set of values that gets copied into the SI. We only use this to communicate *back* to the RM. All other fields are considered incoming fields from the RM into the core. The limited set of fields link the Allocation to an Application, Node and AllocationAsk.

func (*Allocation) SetAllocationID added in v1.5.0 

func (a *Allocation) SetAllocationID(allocationID string)

SetAllocationID set the allocationID for this allocation only for tests

func (*Allocation) SetBindTime added in v1.3.0 

func (a *Allocation) SetBindTime(bindTime time.Time)

func (*Allocation) SetCreateTime added in v1.1.0 

func (a *Allocation) SetCreateTime(createTime time.Time)

func (*Allocation) SetInstanceType added in v1.4.0 

func (a *Allocation) SetInstanceType(instType string)

SetInstanceType sets node instance type for this allocation

func (*Allocation) SetPlaceholderCreateTime added in v1.1.0 

func (a *Allocation) SetPlaceholderCreateTime(placeholderCreateTime time.Time)

SetPlaceholderCreateTime updates the placeholder's creation time

func (*Allocation) SetPlaceholderUsed added in v1.1.0 

func (a *Allocation) SetPlaceholderUsed(placeholderUsed bool)

SetPlaceholderUsed sets whether this alloc is replacing a placeholder

func (*Allocation) SetRelease added in v1.1.0 

func (a *Allocation) SetRelease(release *Allocation)

func (*Allocation) SetReleased added in v1.1.0 

func (a *Allocation) SetReleased(released bool)

SetReleased updates the release status of the allocation

func (*Allocation) SetReservedNodeID added in v1.1.0 

func (a *Allocation) SetReservedNodeID(reservedNodeID string)

SetReservedNodeID sets the node this allocation is reserved for

func (*Allocation) SetResult added in v1.1.0 

func (a *Allocation) SetResult(result AllocationResult)

SetResult sets the result of this allocation

func (*Allocation) String 

func (a *Allocation) String() string

type AllocationAsk 

type AllocationAsk struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewAllocationAsk 

func NewAllocationAsk(allocationKey string, applicationID string, allocatedResource *resources.Resource) *AllocationAsk

func NewAllocationAskFromSI added in v1.1.0 

func NewAllocationAskFromSI(ask *si.AllocationAsk) *AllocationAsk

func (*AllocationAsk) GetAllocatedResource added in v1.1.0 

func (aa *AllocationAsk) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a reference to the allocated resources for this ask. This must be treated as read-only.

func (*AllocationAsk) GetAllocationKey added in v1.1.0 

func (aa *AllocationAsk) GetAllocationKey() string

GetAllocationKey returns the allocation key for this ask

func (*AllocationAsk) GetAllocationLog added in v1.1.0 

func (aa *AllocationAsk) GetAllocationLog() []*AllocationLogEntry

GetAllocationLog returns a list of log entries corresponding to allocation preconditions not being met

func (*AllocationAsk) GetApplicationID added in v1.1.0 

func (aa *AllocationAsk) GetApplicationID() string

GetApplicationID returns the application ID for this ask

func (*AllocationAsk) GetCreateTime 

func (aa *AllocationAsk) GetCreateTime() time.Time

GetCreateTime returns the time this ask was created

func (*AllocationAsk) GetPartitionName added in v1.1.0 

func (aa *AllocationAsk) GetPartitionName() string

GetPartitionName returns the partition name for this ask

func (*AllocationAsk) GetPendingAskRepeat 

func (aa *AllocationAsk) GetPendingAskRepeat() int32

GetPendingAskRepeat gets the number of repeat asks remaining

func (*AllocationAsk) GetPreemptCheckTime added in v1.3.0 

func (aa *AllocationAsk) GetPreemptCheckTime() time.Time

GetPreemptCheckTime returns the time this ask was last evaluated for preemption

func (*AllocationAsk) GetPriority added in v1.1.0 

func (aa *AllocationAsk) GetPriority() int32

GetPriority returns the priority of this ask

func (*AllocationAsk) GetRequiredNode 

func (aa *AllocationAsk) GetRequiredNode() string

GetRequiredNode gets the node (if any) required by this ask.

func (*AllocationAsk) GetTag added in v1.1.0 

func (aa *AllocationAsk) GetTag(tagName string) string

GetTag returns the value of a named tag or an empty string if not present

func (*AllocationAsk) GetTagsClone added in v1.1.0 

func (aa *AllocationAsk) GetTagsClone() map[string]string

GetTagsClone returns the copy of the tags for this ask

func (*AllocationAsk) GetTaskGroup added in v1.1.0 

func (aa *AllocationAsk) GetTaskGroup() string

GetTaskGroup returns the task group name for this ask

func (*AllocationAsk) GetTimeout added in v1.1.0 

func (aa *AllocationAsk) GetTimeout() time.Duration

GetTimeout returns the timeout for this ask

func (*AllocationAsk) HasTriggeredPreemption added in v1.2.0 

func (aa *AllocationAsk) HasTriggeredPreemption() bool

HasTriggeredPreemption returns whether this ask has triggered preemption

func (*AllocationAsk) HasTriggeredScaleUp added in v1.5.0 

func (aa *AllocationAsk) HasTriggeredScaleUp() bool

func (*AllocationAsk) IsAllowPreemptOther added in v1.3.0 

func (aa *AllocationAsk) IsAllowPreemptOther() bool

IsAllowPreemptOther returns whether this ask is allowed to preempt others

func (*AllocationAsk) IsAllowPreemptSelf added in v1.3.0 

func (aa *AllocationAsk) IsAllowPreemptSelf() bool

IsAllowPreemptSelf returns whether preemption is allowed for this ask

func (*AllocationAsk) IsOriginator added in v1.1.0 

func (aa *AllocationAsk) IsOriginator() bool

IsOriginator returns whether this ask is the originator for the application

func (*AllocationAsk) IsPlaceholder added in v1.1.0 

func (aa *AllocationAsk) IsPlaceholder() bool

IsPlaceholder returns whether this ask represents a placeholder

func (*AllocationAsk) IsSchedulingAttempted added in v1.5.0 

func (aa *AllocationAsk) IsSchedulingAttempted() bool

func (*AllocationAsk) LessThan added in v1.4.0 

func (aa *AllocationAsk) LessThan(other *AllocationAsk) bool

func (*AllocationAsk) LogAllocationFailure added in v1.1.0 

func (aa *AllocationAsk) LogAllocationFailure(message string, allocate bool)

LogAllocationFailure keeps track of preconditions not being met for an allocation

func (*AllocationAsk) MarkTriggeredPreemption added in v1.2.0 

func (aa *AllocationAsk) MarkTriggeredPreemption()

MarkTriggeredPreemption marks the current ask because it triggered preemption during allocation

func (*AllocationAsk) SendPredicateFailedEvent added in v1.5.0 

func (aa *AllocationAsk) SendPredicateFailedEvent(message string)

func (*AllocationAsk) SetRequiredNode added in v1.1.0 

func (aa *AllocationAsk) SetRequiredNode(node string)

SetRequiredNode sets the required node (used only by testing so lock is not taken)

func (*AllocationAsk) SetScaleUpTriggered added in v1.5.0 

func (aa *AllocationAsk) SetScaleUpTriggered(triggered bool)

func (*AllocationAsk) SetSchedulingAttempted added in v1.5.0 

func (aa *AllocationAsk) SetSchedulingAttempted(attempted bool)

func (*AllocationAsk) String 

func (aa *AllocationAsk) String() string

func (*AllocationAsk) UpdatePreemptCheckTime added in v1.3.0 

func (aa *AllocationAsk) UpdatePreemptCheckTime()

UpdatePreemptCheckTime is used to mark when this ask is evaluated for preemption

type AllocationLogEntry added in v1.1.0 

type AllocationLogEntry struct {
	Message        string
	LastOccurrence time.Time
	Count          int32
}

type AllocationResult added in v1.1.0 

type AllocationResult int
const (
	None AllocationResult = iota
	Allocated
	AllocatedReserved
	Reserved
	Unreserved
	Replaced
)

func (AllocationResult) String added in v1.1.0 

func (ar AllocationResult) String() string

type Application 

type Application struct {
	ApplicationID  string    // application ID
	Partition      string    // partition Name
	SubmissionTime time.Time // time application was submitted

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewApplication 

func NewApplication(siApp *si.AddApplicationRequest, ugi security.UserGroup, eventHandler handler.EventHandler, rmID string) *Application

func (*Application) AddAllocation 

func (sa *Application) AddAllocation(info *Allocation)

Add a new Allocation to the application

func (*Application) AddAllocationAsk 

func (sa *Application) AddAllocationAsk(ask *AllocationAsk) error

Add an allocation ask to this application If the ask already exist update the existing info

func (*Application) CleanupTrackedResource added in v1.4.0 

func (sa *Application) CleanupTrackedResource()

func (*Application) CurrentState 

func (sa *Application) CurrentState() string

Return the current state or a checked specific state for the application. The state machine handles the locking.

func (*Application) FailApplication 

func (sa *Application) FailApplication(failureMessage string) error

FailApplication fails this application.

func (*Application) FinishedTime 

func (sa *Application) FinishedTime() time.Time

func (*Application) GetAllAllocations 

func (sa *Application) GetAllAllocations() []*Allocation

get a copy of all allocations of the application

func (*Application) GetAllPlaceholderData 

func (sa *Application) GetAllPlaceholderData() []*PlaceholderData

func (*Application) GetAllRequests added in v1.1.0 

func (sa *Application) GetAllRequests() []*AllocationAsk

GetAllRequests returns a copy of all requests of the application

func (*Application) GetAllocatedResource 

func (sa *Application) GetAllocatedResource() *resources.Resource

Return the allocated resources for this application

func (*Application) GetAllocationAsk 

func (sa *Application) GetAllocationAsk(allocationKey string) *AllocationAsk

Return the allocation ask for the key, nil if not found

func (*Application) GetApplicationSummary added in v1.3.0 

func (sa *Application) GetApplicationSummary(rmID string) *ApplicationSummary

func (*Application) GetAskMaxPriority added in v1.2.0 

func (sa *Application) GetAskMaxPriority() int32

func (*Application) GetAskReservations 

func (sa *Application) GetAskReservations(allocKey string) []string

Return the allocation reservations on any node. The returned array is 0 or more keys into the reservations map. No locking must be called while holding the lock

func (*Application) GetMaxAllocatedResource 

func (sa *Application) GetMaxAllocatedResource() *resources.Resource

func (*Application) GetPendingResource 

func (sa *Application) GetPendingResource() *resources.Resource

Return the pending resources for this application

func (*Application) GetPlaceholderAsk 

func (sa *Application) GetPlaceholderAsk() *resources.Resource

Return the total placeholder ask for this application Is only set on app creation and used when app is added to a queue

func (*Application) GetPlaceholderResource 

func (sa *Application) GetPlaceholderResource() *resources.Resource

Return the allocated placeholder resources for this application

func (*Application) GetQueue 

func (sa *Application) GetQueue() *Queue

func (*Application) GetQueuePath 

func (sa *Application) GetQueuePath() string

func (*Application) GetRejectedMessage 

func (sa *Application) GetRejectedMessage() string

func (*Application) GetReservations 

func (sa *Application) GetReservations() []string

GetReservations returns an array of all reservation keys for the application. This will return an empty array if there are no reservations. Visible for tests

func (*Application) GetStateLog 

func (sa *Application) GetStateLog() []*StateLogEntry

func (*Application) GetTag 

func (sa *Application) GetTag(tag string) string

Get a tag from the application Note: tags are not case sensitive

func (*Application) GetUser 

func (sa *Application) GetUser() security.UserGroup

get a copy of the user details for the application

func (*Application) HandleApplicationEvent 

func (sa *Application) HandleApplicationEvent(event applicationEvent) error

HandleApplicationEvent handles the state event for the application. The application lock is expected to be held.

func (*Application) HandleApplicationEventWithInfo 

func (sa *Application) HandleApplicationEventWithInfo(event applicationEvent, eventInfo string) error

HandleApplicationEventWithInfo handles the state event for the application with associated info object. The application lock is expected to be held.

func (*Application) HasPlaceholderAllocation added in v1.4.0 

func (sa *Application) HasPlaceholderAllocation() bool

func (*Application) HasReserved added in v1.2.0 

func (sa *Application) HasReserved() bool

HasReserved returns true if the application has any reservations.

func (*Application) IsAccepted 

func (sa *Application) IsAccepted() bool

func (*Application) IsAllocationAssignedToApp 

func (sa *Application) IsAllocationAssignedToApp(alloc *Allocation) bool

func (*Application) IsCompleted 

func (sa *Application) IsCompleted() bool

func (*Application) IsCompleting 

func (sa *Application) IsCompleting() bool

func (*Application) IsCreateForced added in v1.4.0 

func (sa *Application) IsCreateForced() bool

func (*Application) IsExpired 

func (sa *Application) IsExpired() bool

func (*Application) IsFailed 

func (sa *Application) IsFailed() bool

func (*Application) IsFailing 

func (sa *Application) IsFailing() bool

func (*Application) IsNew 

func (sa *Application) IsNew() bool

func (*Application) IsRejected 

func (sa *Application) IsRejected() bool

func (*Application) IsReservedOnNode 

func (sa *Application) IsReservedOnNode(nodeID string) bool

IsReservedOnNode returns true if and only if the node has been reserved by the application An empty nodeID is never reserved.

func (*Application) IsResuming 

func (sa *Application) IsResuming() bool

func (*Application) IsRunning 

func (sa *Application) IsRunning() bool

func (*Application) IsStarting 

func (sa *Application) IsStarting() bool

func (*Application) LogAppSummary added in v1.3.0 

func (sa *Application) LogAppSummary(rmID string)

func (*Application) OnStateChange 

func (sa *Application) OnStateChange(event *fsm.Event, eventInfo string)

OnStatChange every time the application enters a new state. It sends an event about the state change to the shim as an application update. The only state that does not generate an event is Rejected.

func (*Application) RecoverAllocationAsk 

func (sa *Application) RecoverAllocationAsk(ask *AllocationAsk)

Add the ask when a node allocation is recovered. Maintaining the rule that an Allocation always has a link to an AllocationAsk. Safeguarded against a nil but the recovery generates the ask and should never be nil.

func (*Application) RejectApplication 

func (sa *Application) RejectApplication(rejectedMessage string) error

RejectApplication rejects this application.

func (*Application) RemoveAllAllocations 

func (sa *Application) RemoveAllAllocations() []*Allocation

Remove all allocations from the application. All allocations that have been removed are returned.

func (*Application) RemoveAllocation 

func (sa *Application) RemoveAllocation(allocationID string, releaseType si.TerminationType) *Allocation

Remove the Allocation from the application. Return the allocation that was removed or nil if not found.

func (*Application) RemoveAllocationAsk 

func (sa *Application) RemoveAllocationAsk(allocKey string) int

Remove one or more allocation asks from this application. This also removes any reservations that are linked to the ask. The return value is the number of reservations released

func (*Application) ReplaceAllocation 

func (sa *Application) ReplaceAllocation(allocationID string) *Allocation

func (*Application) Reserve 

func (sa *Application) Reserve(node *Node, ask *AllocationAsk) error

Reserve the application for this node and ask combination. If the reservation fails the function returns false, if the reservation is made it returns true. If the node and ask combination was already reserved for the application this is a noop and returns true.

func (*Application) SetQueue 

func (sa *Application) SetQueue(queue *Queue)

Set the leaf queue the application runs in.

func (*Application) SetQueuePath 

func (sa *Application) SetQueuePath(queuePath string)

Set the leaf queue the application runs in. The queue will be created when the app is added to the partition. The queue name is set to what the placement rule returned.

func (*Application) SetState 

func (sa *Application) SetState(state string)

func (*Application) SetTerminatedCallback 

func (sa *Application) SetTerminatedCallback(callback func(appID string))

func (*Application) SetTimedOutPlaceholder added in v1.1.0 

func (sa *Application) SetTimedOutPlaceholder(taskGroupName string, timedOut int64)

SetTimedOutPlaceholder should be used for testing only.

func (*Application) StartTime added in v1.3.0 

func (sa *Application) StartTime() time.Time

func (*Application) String 

func (sa *Application) String() string

func (*Application) UnReserve 

func (sa *Application) UnReserve(node *Node, ask *AllocationAsk) (int, error)

UnReserve the application for this node and ask combination. This first removes the reservation from the node. If the reservation does not exist it returns 0 for reservations removed, if the reservation is removed it returns 1. The error is set if the reservation key cannot be removed from the app or node.

func (*Application) UnSetQueue 

func (sa *Application) UnSetQueue()

remove the leaf queue the application runs in, used when completing the app

func (*Application) UpdateAskRepeat 

func (sa *Application) UpdateAskRepeat(allocKey string, delta int32) (*resources.Resource, error)

type ApplicationSummary added in v1.3.0 

type ApplicationSummary struct {
	ApplicationID       string
	SubmissionTime      time.Time
	StartTime           time.Time
	FinishTime          time.Time
	User                string
	Queue               string
	State               string
	RmID                string
	ResourceUsage       *resources.TrackedResource
	PreemptedResource   *resources.TrackedResource
	PlaceholderResource *resources.TrackedResource
}

func (*ApplicationSummary) DoLogging added in v1.3.0 

func (as *ApplicationSummary) DoLogging()

func (*ApplicationSummary) String added in v1.5.0 

func (as *ApplicationSummary) String() string

type Node 

type Node struct {
	// Fields for fast access These fields are considered read only.
	// Values should only be set when creating a new node and never changed.
	NodeID    string
	Hostname  string
	Rackname  string
	Partition string

	locking.RWMutex
	// contains filtered or unexported fields
}

func NewNode 

func NewNode(proto *si.NodeInfo) *Node

func (*Node) AddAllocation 

func (sn *Node) AddAllocation(alloc *Allocation) bool

AddAllocation adds the allocation to the node. Used resources will increase available will decrease. A nil Allocation makes no changes. Preempted resources must have been released already. Do a sanity check to make sure it still fits in the node and nothing has changed

func (*Node) AddListener 

func (sn *Node) AddListener(listener NodeListener)

func (*Node) CanAllocate 

func (sn *Node) CanAllocate(res *resources.Resource) bool

CanAllocate checks if the proposed allocation fits in the available resources. If the proposed allocation does not fit false is returned.

func (*Node) FitInNode 

func (sn *Node) FitInNode(resRequest *resources.Resource) bool

FitInNode checks if the request fits in the node. All resources types requested must match the resource types provided by the nodes. A request may ask for only a subset of the types, but the node must provide at least the resource types requested in a larger or equal quantity as requested.

func (*Node) GetAllAllocations 

func (sn *Node) GetAllAllocations() []*Allocation

Get a copy of the allocations on this node

func (*Node) GetAllocatedResource 

func (sn *Node) GetAllocatedResource() *resources.Resource

Get the allocated resource on this node.

func (*Node) GetAllocation 

func (sn *Node) GetAllocation(allocationID string) *Allocation

Return the allocation based on the allocationID of the allocation. returns nil if the allocation is not found

func (*Node) GetAttribute 

func (sn *Node) GetAttribute(key string) string

Get an attribute by name. The most used attributes can be directly accessed via the fields: HostName, RackName and Partition. This is a lock free call. All attributes are considered read only

func (*Node) GetAttributes added in v1.3.0 

func (sn *Node) GetAttributes() map[string]string

func (*Node) GetAvailableResource 

func (sn *Node) GetAvailableResource() *resources.Resource

Get the available resource on this node.

func (*Node) GetCapacity 

func (sn *Node) GetCapacity() *resources.Resource

func (*Node) GetFitInScoreForAvailableResource added in v1.3.0 

func (sn *Node) GetFitInScoreForAvailableResource(res *resources.Resource) float64

GetFitInScoreForAvailableResource calculates a fit in score for "res" based on the current available resources, avoiding cloning. The caller must ensure that "res" cannot change while this method is running.

func (*Node) GetInstanceType added in v1.3.0 

func (sn *Node) GetInstanceType() string

Get InstanceType of this node. This is a lock free call because all attributes are considered read only

func (*Node) GetOccupiedResource 

func (sn *Node) GetOccupiedResource() *resources.Resource

func (*Node) GetReservationKeys added in v1.2.0 

func (sn *Node) GetReservationKeys() []string

GetReservationKeys Return an array of all reservation keys for the node. This will return an empty array if there are no reservations. Visible for tests

func (*Node) GetReservations 

func (sn *Node) GetReservations() []*reservation

GetReservations returns all reservation made on this node

func (*Node) GetResourceUsageShares 

func (sn *Node) GetResourceUsageShares() map[string]float64

GetResourceUsageShares gets a map of name -> resource usages per type in shares (0 to 1). Can return NaN.

func (*Node) GetUtilizedResource 

func (sn *Node) GetUtilizedResource() *resources.Resource

Get the utilized resource on this node.

func (*Node) IsReady 

func (sn *Node) IsReady() bool

func (*Node) IsReserved 

func (sn *Node) IsReserved() bool

Return if the node has been reserved by any application

func (*Node) IsSchedulable 

func (sn *Node) IsSchedulable() bool

Can this node be used in scheduling.

func (*Node) RemoveAllocation 

func (sn *Node) RemoveAllocation(allocationID string) *Allocation

Remove the allocation to the node. Returns nil if the allocation was not found and no changes are made. If the allocation is found the Allocation removed is returned. Used resources will decrease available will increase as per the allocation removed.

func (*Node) RemoveListener 

func (sn *Node) RemoveListener(listener NodeListener)

func (*Node) ReplaceAllocation 

func (sn *Node) ReplaceAllocation(allocationID string, replace *Allocation, delta *resources.Resource)

ReplaceAllocation replaces the placeholder with the real allocation on the node. The delta passed in is the difference in resource usage between placeholder and real allocation. It should always be a negative value or zero: it is a decrease in usage or no change

func (*Node) Reserve 

func (sn *Node) Reserve(app *Application, ask *AllocationAsk) error

Reserve the node for this application and ask combination, if not reserved yet. The reservation is checked against the node resources. If the reservation fails the function returns false, if the reservation is made it returns true.

func (*Node) SendNodeAddedEvent added in v1.4.0 

func (sn *Node) SendNodeAddedEvent()

func (*Node) SendNodeRemovedEvent added in v1.4.0 

func (sn *Node) SendNodeRemovedEvent()

func (*Node) SetCapacity 

func (sn *Node) SetCapacity(newCapacity *resources.Resource) *resources.Resource

func (*Node) SetOccupiedResource 

func (sn *Node) SetOccupiedResource(occupiedResource *resources.Resource)

func (*Node) SetReady 

func (sn *Node) SetReady(ready bool)

func (*Node) SetSchedulable 

func (sn *Node) SetSchedulable(schedulable bool)

Set the node to unschedulable. This will cause the node to be skipped during the scheduling cycle. Visible for testing only

func (*Node) String 

func (sn *Node) String() string

type NodeCollection 

type NodeCollection interface {
	AddNode(node *Node) error
	RemoveNode(nodeID string) *Node
	GetNode(nodeID string) *Node
	GetNodeCount() int
	GetNodes() []*Node
	GetNodeIterator() NodeIterator
	GetFullNodeIterator() NodeIterator
	SetNodeSortingPolicy(policy NodeSortingPolicy)
	GetNodeSortingPolicy() NodeSortingPolicy
}

NodeCollection represents a collection of nodes for a partition. Implementations of this interface must be internally synchronized to avoid data races.

func NewNodeCollection 

func NewNodeCollection(partition string) NodeCollection

Create a new collection for the given partition.

type NodeIterator 

type NodeIterator interface {
	// ForEachNode Calls the provided function on the sorted Node object until it returns false
	ForEachNode(func(*Node) bool)
}

NodeIterator iterates over a list of nodes based on the defined policy

type NodeListener 

type NodeListener interface {
	NodeUpdated(sn *Node)
}

type NodeSortingPolicy 

type NodeSortingPolicy interface {
	PolicyType() policies.SortingPolicy
	ScoreNode(node *Node) float64
	ResourceWeights() map[string]float64
}

func NewNodeSortingPolicy 

func NewNodeSortingPolicy(policyType string, resourceWeights map[string]float64) NodeSortingPolicy

type ObjectEvent 

type ObjectEvent int

---------------------------------- object events these events are used for: partitions and managed queues ---------------------------------- 

const (
	Remove ObjectEvent = iota
	Start
	Stop
)

func (ObjectEvent) String 

func (oe ObjectEvent) String() string

type ObjectState 

type ObjectState int

---------------------------------- object states these states are used by: partitions and managed queues ---------------------------------- 

const (
	Active ObjectState = iota
	Draining
	Stopped
)

func (ObjectState) String 

func (os ObjectState) String() string

type PlaceholderData 

type PlaceholderData struct {
	TaskGroupName string
	Count         int64
	MinResource   *resources.Resource
	Replaced      int64
	TimedOut      int64
}

type PreemptionContext added in v1.1.0 

type PreemptionContext struct {
	locking.RWMutex
	// contains filtered or unexported fields
}

func NewRequiredNodePreemptor added in v1.2.0 

func NewRequiredNodePreemptor(node *Node, requiredAsk *AllocationAsk) *PreemptionContext

func (*PreemptionContext) GetVictims added in v1.1.0 

func (p *PreemptionContext) GetVictims() []*Allocation

type Preemptor added in v1.3.0 

type Preemptor struct {
	// contains filtered or unexported fields
}

Preemptor encapsulates the functionality required for preemption victim selection

func NewPreemptor added in v1.3.0 

func NewPreemptor(application *Application, headRoom *resources.Resource, preemptionDelay time.Duration, ask *AllocationAsk, iterator NodeIterator, nodesTried bool) *Preemptor

NewPreemptor creates a new preemptor. The preemptor itself is not thread safe, and assumes the application lock is held.

func (*Preemptor) CheckPreconditions added in v1.3.0 

func (p *Preemptor) CheckPreconditions() bool

CheckPreconditions performs simple sanity checks designed to determine if preemption should be attempted for an ask. If checks succeed, updates the ask preemption check time.

func (*Preemptor) TryPreemption added in v1.3.0 

func (p *Preemptor) TryPreemption() (*Allocation, bool)

type Queue 

type Queue struct {
	QueuePath string // Fully qualified path for the queue
	Name      string // Queue name as in the config etc.

	locking.RWMutex
	// contains filtered or unexported fields
}

Queue structure inside Scheduler

func NewConfiguredQueue 

func NewConfiguredQueue(conf configs.QueueConfig, parent *Queue) (*Queue, error)

NewConfiguredQueue creates a new queue from scratch based on the configuration lock free as it cannot be referenced yet

func NewDynamicQueue 

func NewDynamicQueue(name string, leaf bool, parent *Queue) (*Queue, error)

NewDynamicQueue creates a new queue to be added to the system based on the placement rules A dynamically added queue can never be the root queue so parent must be set lock free as it cannot be referenced yet

func NewRecoveryQueue added in v1.4.0 

func NewRecoveryQueue(parent *Queue) (*Queue, error)

NewRecoveryQueue creates a recovery queue if it does not exist. The recovery queue is a dynamic queue, but has an invalid name so that it cannot be directly referenced.

func (*Queue) AddApplication 

func (sq *Queue) AddApplication(app *Application)

AddApplication adds the application to the queue. All checks are assumed to have passed before we get here. No update of pending resource is needed as it should not have any requests yet. Replaces the existing application without further checks.

func (*Queue) ApplyConf 

func (sq *Queue) ApplyConf(conf configs.QueueConfig) error

ApplyConf is the locked version of applyConf

func (*Queue) CheckAdminAccess 

func (sq *Queue) CheckAdminAccess(user security.UserGroup) bool

CheckAdminAccess checks if the user has access to the queue to perform administrative actions. The check is performed recursively: i.e. access to the parent allows access to this queue.

func (*Queue) CheckSubmitAccess 

func (sq *Queue) CheckSubmitAccess(user security.UserGroup) bool

CheckSubmitAccess checks if the user has access to the queue to submit an application. The check is performed recursively: i.e. access to the parent allows access to this queue. This will check both submitACL and adminACL.

func (*Queue) CurrentState 

func (sq *Queue) CurrentState() string

CurrentState returns the current state of the queue in string form.

func (*Queue) DecAllocatedResource 

func (sq *Queue) DecAllocatedResource(alloc *resources.Resource) error

DecAllocatedResource decrement the allocated resources for this queue (recursively) Guard against going below zero resources.

func (*Queue) DecPreemptingResource added in v1.3.0 

func (sq *Queue) DecPreemptingResource(alloc *resources.Resource)

DecPreemptingResource decrements the preempting resources for this queue (recursively).

func (*Queue) FindEligiblePreemptionVictims added in v1.3.0 

func (sq *Queue) FindEligiblePreemptionVictims(queuePath string, ask *AllocationAsk) map[string]*QueuePreemptionSnapshot

FindEligiblePreemptionVictims is used to locate tasks which may be preempted for the given ask. queuePath is the fully-qualified path of the queue where ask resides ask is the ask we are attempting to preempt for return is a map of potential victims keyed by queue path

func (*Queue) FindQueueByAppID added in v1.3.0 

func (sq *Queue) FindQueueByAppID(appID string) *Queue

FindQueueByAppID searches the queue hierarchy for an application with the given appID and returns the queue it belongs to

func (*Queue) GetActualGuaranteedResource added in v1.3.0 

func (sq *Queue) GetActualGuaranteedResource() *resources.Resource

GetActualGuaranteedResources returns the actual (including parent) guaranteed resources for the queue.

func (*Queue) GetAllocatedResource 

func (sq *Queue) GetAllocatedResource() *resources.Resource

GetAllocatedResource returns a clone of the allocated resources for this queue.

func (*Queue) GetApplication added in v1.1.0 

func (sq *Queue) GetApplication(appID string) *Application

getApplication return the Application based on the ID.

func (*Queue) GetChildQueue 

func (sq *Queue) GetChildQueue(name string) *Queue

GetChildQueue returns a queue if the name exists in the child map as a key.

func (*Queue) GetCopyOfApps 

func (sq *Queue) GetCopyOfApps() map[string]*Application

GetCopyOfApps gets a shallow copy of all non-completed apps holding the lock

func (*Queue) GetCopyOfChildren 

func (sq *Queue) GetCopyOfChildren() map[string]*Queue

GetCopyOfChildren return a shallow copy of the child queue map. This is used by the partition manager to find all queues to clean however we can not guarantee that there is no new child added while we clean up since there is no overall lock on the scheduler. We'll need to test just before to make sure the parent is empty

func (*Queue) GetCurrentPriority added in v1.2.0 

func (sq *Queue) GetCurrentPriority() int32

func (*Queue) GetGuaranteedResource 

func (sq *Queue) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource returns a clone of the guaranteed resource for the queue.

func (*Queue) GetMaxQueueSet 

func (sq *Queue) GetMaxQueueSet() *resources.Resource

GetMaxQueueSet returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The cluster size, which defines the root limit, is not relevant for this call. Contrary to the GetMaxResource call. This will return nil unless a limit is set. Used during scheduling in an auto-scaling cluster. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetMaxResource 

func (sq *Queue) GetMaxResource() *resources.Resource

GetMaxResource returns the max resource for the queue. The max resource should never be larger than the max resource of the parent. The root queue always has its limit set to the total cluster size (dynamic based on node registration) In case there are no nodes in a newly started cluster and no queues have a limit configured this call will return nil. NOTE: if a resource quantity is missing and a limit is defined the missing quantity will be seen as a limit of 0. When defining a limit you therefore should define all resource quantities.

func (*Queue) GetPartitionQueueDAOInfo 

func (sq *Queue) GetPartitionQueueDAOInfo(exclude bool) dao.PartitionQueueDAOInfo

GetPartitionQueueDAOInfo returns the queue hierarchy as an object for a REST call. Exclude is true, which means that returns the specified queue object, but does not return the children of the specified queue.

func (*Queue) GetPendingResource 

func (sq *Queue) GetPendingResource() *resources.Resource

GetPendingResource returns the pending resources for this queue.

func (*Queue) GetPreemptingResource 

func (sq *Queue) GetPreemptingResource() *resources.Resource

GetPreemptingResource returns a clone of the preempting resources for this queue.

func (*Queue) GetPreemptionDelay added in v1.3.0 

func (sq *Queue) GetPreemptionDelay() time.Duration

func (*Queue) GetPreemptionPolicy added in v1.3.0 

func (sq *Queue) GetPreemptionPolicy() policies.PreemptionPolicy

func (*Queue) GetPriorityPolicyAndOffset added in v1.3.0 

func (sq *Queue) GetPriorityPolicyAndOffset() (policies.PriorityPolicy, int32)

func (*Queue) GetQueueOutstandingRequests 

func (sq *Queue) GetQueueOutstandingRequests(total *[]*AllocationAsk)

GetQueueOutstandingRequests builds a slice of pending allocation asks that fits into the queue's headroom.

func (*Queue) GetQueuePath 

func (sq *Queue) GetQueuePath() string

GetQueuePath returns the fully qualified path of this queue.

func (*Queue) GetReservedApps added in v1.2.0 

func (sq *Queue) GetReservedApps() map[string]int

GetReservedApps returns a shallow copy of the reserved app list locked to prevent race conditions from event updates

func (*Queue) IncAllocatedResource 

func (sq *Queue) IncAllocatedResource(alloc *resources.Resource, nodeReported bool) error

IncAllocatedResource increments the allocated resources for this queue (recursively). Guard against going over max resources if set

func (*Queue) IncPreemptingResource 

func (sq *Queue) IncPreemptingResource(alloc *resources.Resource)

IncPreemptingResource increments the preempting resources for this queue (recursively).

func (*Queue) IsDraining 

func (sq *Queue) IsDraining() bool

IsDraining returns true if the queue in Draining state. Existing applications will still be scheduled No new applications will be accepted.

func (*Queue) IsEmpty 

func (sq *Queue) IsEmpty() bool

IsEmpty returns true if a queue is empty based on the following definition: A parent queue is empty when it has no children left A leaf queue is empty when there are no applications left

func (*Queue) IsLeafQueue 

func (sq *Queue) IsLeafQueue() bool

IsLeafQueue returns true is the queue a leaf. Returns false for a parent queue.

func (*Queue) IsManaged 

func (sq *Queue) IsManaged() bool

IsManaged returns true for a managed queue. Returns false for a dynamic queue.

func (*Queue) IsPrioritySortEnabled added in v1.2.0 

func (sq *Queue) IsPrioritySortEnabled() bool

func (*Queue) IsRunning 

func (sq *Queue) IsRunning() bool

IsRunning returns true if the queue in Active state.

func (*Queue) IsStopped 

func (sq *Queue) IsStopped() bool

IsStopped returns true if the queue in Stopped state. The queue is skipped for scheduling in this state.

func (*Queue) MarkQueueForRemoval 

func (sq *Queue) MarkQueueForRemoval()

MarkQueueForRemoval marks the managed queue for removal from the system. This can be executed multiple times and is only effective the first time. This is a noop on an unmanaged queue.

func (*Queue) RemoveApplication 

func (sq *Queue) RemoveApplication(app *Application)

RemoveApplication removes the app from the list of tracked applications. Make sure that the app is assigned to this queue and not removed yet. If not found this call is a noop

func (*Queue) RemoveQueue 

func (sq *Queue) RemoveQueue() bool

RemoveQueue remove the queue from the structure. Since nothing is allocated there shouldn't be anything referencing this queue anymore. The real removal is the removal of the queue from the parent's child list. Use a read lock on this queue to prevent other changes but allow status checks etc.

func (*Queue) Reserve 

func (sq *Queue) Reserve(appID string)

Reserve increments the number of reservations for the application adding it to the map if needed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) SetMaxResource 

func (sq *Queue) SetMaxResource(max *resources.Resource)

SetMaxResource sets the max resource for the root queue. Called as part of adding or removing a node. Should only happen on the root, all other queues get it from the config via properties.

func (*Queue) SetMaxRunningApps added in v1.2.0 

func (sq *Queue) SetMaxRunningApps(max int)

SetMaxRunningApps allows setting the maximum running apps on a queue test only

func (*Queue) String 

func (sq *Queue) String() string

func (*Queue) SupportTaskGroup 

func (sq *Queue) SupportTaskGroup() bool

SupportTaskGroup returns true if the queue supports task groups. FIFO and StateAware sorting policies can support this. NOTE: this call does not make sense for a parent queue, and always returns false

func (*Queue) TryAllocate 

func (sq *Queue) TryAllocate(iterator func() NodeIterator, fullIterator func() NodeIterator, getnode func(string) *Node, allowPreemption bool) *Allocation

TryAllocate tries to allocate a pending requests. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryPlaceholderAllocate 

func (sq *Queue) TryPlaceholderAllocate(iterator func() NodeIterator, getnode func(string) *Node) *Allocation

TryPlaceholderAllocate tries to replace a placeholders with a real allocation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are sorted based on the application sortPolicy. Applications without pending resources are skipped. Lock free call this all locks are taken when needed in called functions

func (*Queue) TryReservedAllocate 

func (sq *Queue) TryReservedAllocate(iterator func() NodeIterator) *Allocation

TryReservedAllocate tries to allocate a reservation. This only gets called if there is a pending request on this queue or its children. This is a depth first algorithm: descend into the depth of the queue tree first. Child queues are sorted based on the configured queue sortPolicy. Queues without pending resources are skipped. Applications are currently NOT sorted and are iterated over in a random order. Lock free call this all locks are taken when needed in called functions

func (*Queue) UnReserve 

func (sq *Queue) UnReserve(appID string, releases int)

UnReserve decrements the number of reservations for the application removing it to the map if all reservations are removed. No checks this is only called when a reservation is processed using the app stored in the queue.

func (*Queue) UpdateApplicationPriority added in v1.2.0 

func (sq *Queue) UpdateApplicationPriority(applicationID string, priority int32)

func (*Queue) UpdateQueuePriority added in v1.2.0 

func (sq *Queue) UpdateQueuePriority(queueName string, priority int32)

func (*Queue) UpdateQueueProperties added in v1.2.0 

func (sq *Queue) UpdateQueueProperties()

UpdateQueueProperties updates the queue properties defined as text

type QueuePreemptionSnapshot added in v1.3.0 

type QueuePreemptionSnapshot struct {
	Parent             *QueuePreemptionSnapshot // snapshot of parent queue
	QueuePath          string                   // fully qualified path to queue
	Leaf               bool                     // true if queue is a leaf queue
	AllocatedResource  *resources.Resource      // allocated resources
	PreemptingResource *resources.Resource      // resources currently flagged for preemption
	MaxResource        *resources.Resource      // maximum resources for this queue
	GuaranteedResource *resources.Resource      // guaranteed resources for this queue
	PotentialVictims   []*Allocation            // list of allocations which could be preempted
}

QueuePreemptionSnapshot is used to track a snapshot of a queue for preemption

func (*QueuePreemptionSnapshot) AddAllocation added in v1.3.0 

func (qps *QueuePreemptionSnapshot) AddAllocation(alloc *resources.Resource)

AddAllocation adds an allocation to this snapshot's resource usage

func (*QueuePreemptionSnapshot) Duplicate added in v1.3.0 

func (qps *QueuePreemptionSnapshot) Duplicate(copy map[string]*QueuePreemptionSnapshot) *QueuePreemptionSnapshot

Duplicate creates a copy of this snapshot into the given map by queue path

func (*QueuePreemptionSnapshot) GetGuaranteedResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) GetGuaranteedResource() *resources.Resource

GetGuaranteedResource computes the current guaranteed resources considering parent guaranteed

func (*QueuePreemptionSnapshot) GetMaxResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) GetMaxResource() *resources.Resource

GetMaxResource computes the current max resources considering parent max

func (*QueuePreemptionSnapshot) GetRemainingGuaranteed added in v1.3.0 

func (qps *QueuePreemptionSnapshot) GetRemainingGuaranteed() *resources.Resource

func (*QueuePreemptionSnapshot) IsAtOrAboveGuaranteedResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) IsAtOrAboveGuaranteedResource() bool

IsAtOrAboveGuaranteedResource determines if this queue is exceeding resource guarantees and therefore may be eligible for further preemption

func (*QueuePreemptionSnapshot) IsWithinGuaranteedResource added in v1.3.0 

func (qps *QueuePreemptionSnapshot) IsWithinGuaranteedResource() bool

IsWithinGuaranteedResource determines if this queue is within its current resource guarantees

func (*QueuePreemptionSnapshot) RemoveAllocation added in v1.3.0 

func (qps *QueuePreemptionSnapshot) RemoveAllocation(alloc *resources.Resource)

RemoveAllocation removes an allocation from this snapshot's resource usage

type StateLogEntry 

type StateLogEntry struct {
	Time             time.Time
	ApplicationState string
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.