Documentation
¶
Overview ¶
Package scheduler orders tasks in distro queues.
Index ¶
- Constants
- func AlternateTaskFinder(ctx context.Context, d distro.Distro) ([]task.Task, error)
- func FilterTasksWithVersionCache(tasks []task.Task) ([]task.Task, map[string]model.Version, error)
- func GetDistroQueueInfo(distroID string, tasks []task.Task, maxDurationThreshold time.Duration, ...) model.DistroQueueInfo
- func LegacyFindRunnableTasks(ctx context.Context, d distro.Distro) ([]task.Task, error)
- func ParallelTaskFinder(ctx context.Context, d distro.Distro) ([]task.Task, error)
- func PersistTaskQueue(distro string, tasks []task.Task, distroQueueInfo model.DistroQueueInfo) error
- func PlanDistro(ctx context.Context, conf Configuration, s *evergreen.Settings) error
- func PopulateCaches(id string, tasks []task.Task) ([]task.Task, error)
- func PrioritizeTasks(d *distro.Distro, tasks []task.Task, opts TaskPlannerOptions) ([]task.Task, error)
- func RunnableTasksPipeline(ctx context.Context, d distro.Distro) ([]task.Task, error)
- func SpawnHosts(ctx context.Context, d distro.Distro, newHostsNeeded int, ...) ([]host.Host, error)
- func UpdateStaticDistro(ctx context.Context, d distro.Distro) error
- func UtilizationBasedHostAllocator(ctx context.Context, hostAllocatorData *HostAllocatorData) (int, int, error)
- type CmpBasedTaskComparator
- type CmpBasedTaskPrioritizer
- type CmpBasedTaskQueues
- type Configuration
- type HostAllocator
- type HostAllocatorData
- type Scheduler
- type StringSet
- type TaskFinder
- type TaskGroupData
- type TaskList
- type TaskPlan
- type TaskPlanner
- type TaskPlannerOptions
- type TaskPrioritizer
- type Unit
- type UnitCache
Constants ¶
const RunnerName = "scheduler"
Variables ¶
This section is empty.
Functions ¶
func AlternateTaskFinder ¶
func FilterTasksWithVersionCache ¶
FilterTasksWithVersionCache finds tasks whose versions have already been created, and returns those tasks, as well as a map of version IDs to versions.
func GetDistroQueueInfo ¶
func GetDistroQueueInfo(distroID string, tasks []task.Task, maxDurationThreshold time.Duration, opts TaskPlannerOptions) model.DistroQueueInfo
Returns the distroQueueInfo for the given set of tasks having set the task.ExpectedDuration for each task.
func LegacyFindRunnableTasks ¶
The old Task finderDBTaskFinder, with the dependency check implemented in Go, instead of using $graphLookup
func ParallelTaskFinder ¶
func PersistTaskQueue ¶
func PersistTaskQueue(distro string, tasks []task.Task, distroQueueInfo model.DistroQueueInfo) error
PersistTaskQueue saves the task queue to the database. Returns an error if the db call returns an error.
func PlanDistro ¶
func PopulateCaches ¶
PopulateCaches runs setup functions and is used by the new/tunable scheduler to reprocess tasks before running the new planner.
func PrioritizeTasks ¶
func RunnableTasksPipeline ¶
func SpawnHosts ¶
func SpawnHosts(ctx context.Context, d distro.Distro, newHostsNeeded int, pool *evergreen.ContainerPool) ([]host.Host, error)
Call out to the embedded Manager to spawn hosts. Takes in a map of distro -> number of hosts to spawn for the distro. Returns a map of distro -> hosts spawned, and an error if one occurs. The pool parameter is assumed to be the one from the distro passed in
Types ¶
type CmpBasedTaskComparator ¶
type CmpBasedTaskComparator struct {
// contains filtered or unexported fields
}
CmpBasedTaskComparator runs the tasks through a slice of comparator functions determining which is more important.
func NewCmpBasedTaskComparator ¶
func NewCmpBasedTaskComparator(id string) *CmpBasedTaskComparator
NewCmpBasedTaskComparator returns a new task prioritizer, using the default set of comparators as well as the setup functions necessary for those comparators.
func (*CmpBasedTaskComparator) Len ¶
func (cbtc *CmpBasedTaskComparator) Len() int
func (*CmpBasedTaskComparator) Less ¶
func (cbtc *CmpBasedTaskComparator) Less(i, j int) bool
func (*CmpBasedTaskComparator) Swap ¶
func (cbtc *CmpBasedTaskComparator) Swap(i, j int)
type CmpBasedTaskPrioritizer ¶
type CmpBasedTaskPrioritizer struct {
// contains filtered or unexported fields
}
func (*CmpBasedTaskPrioritizer) PrioritizeTasks ¶
func (prioritizer *CmpBasedTaskPrioritizer) PrioritizeTasks(distroId string, tasks []task.Task, versions map[string]model.Version) ([]task.Task, map[string]map[string]string, error)
PrioritizeTask prioritizes the tasks to run. First splits the tasks into slices based on whether they are part of patch versions or automatically created versions. Then prioritizes each slice, and merges them. Returns a full slice of the prioritized tasks, and an error if one occurs.
type CmpBasedTaskQueues ¶
type CmpBasedTaskQueues struct {
HighPriorityTasks []task.Task
PatchTasks []task.Task
RepotrackerTasks []task.Task
}
CmpBasedTaskQueues represents the three types of queues that are created for merging together into one queue. The HighPriorityTasks list represent the tasks that are always placed at the front of the queue PatchTasks and RepotrackerTasks are interleaved after the high priority tasks.
type Configuration ¶
type HostAllocator ¶
type HostAllocator func(context.Context, *HostAllocatorData) (newHostsNeeded int, estimatedFreeHosts int, err error)
HostAllocator is responsible for determining how many new hosts should be spun up. The first returned int is this number, and the second returned int is the rough number of free hosts.
func GetHostAllocator ¶
func GetHostAllocator(name string) HostAllocator
type HostAllocatorData ¶
type HostAllocatorData struct {
Distro distro.Distro
ExistingHosts []host.Host
UsesContainers bool
ContainerPool *evergreen.ContainerPool
DistroQueueInfo model.DistroQueueInfo
}
type Scheduler ¶
type Scheduler struct {
*evergreen.Settings
TaskPrioritizer
HostAllocator
FindRunnableTasks TaskFinder
}
Responsible for prioritizing and scheduling tasks to be run, on a per-distro basis.
type StringSet ¶
type StringSet map[string]struct{}
StringSet provides simple tools for managing sets of strings.
type TaskFinder ¶
func GetTaskFinder ¶
func GetTaskFinder(version string) TaskFinder
type TaskGroupData ¶
type TaskGroupData struct {
Hosts []host.Host
Info model.TaskGroupInfo
}
type TaskList ¶
TaskList implements sort.Interface on top of a slice of tasks. The provided sorting, orders members of task groups, and then prioritizes tasks by the number of dependencies, priority, and expected duration. This sorting is used for ordering tasks within a unit.
type TaskPlan ¶
type TaskPlan []*Unit
TaskPlan provides a sortable interface on top of a slice of schedulable units, with ordering of units provided by the implementation of RankValue.
func PrepareTasksForPlanning ¶
PrepareTasksForPlanning takes a list of tasks for a distro and returns a TaskPlan, grouping tasks into the appropriate units.
type TaskPlanner ¶
type TaskPlannerOptions ¶
type TaskPrioritizer ¶
type TaskPrioritizer interface {
// Takes in a slice of tasks and the current MCI settings.
// Returns the slice of tasks, sorted in the order in which they should
// be run, as well as an error if appropriate.
PrioritizeTasks(distroId string, tasks []task.Task, versions map[string]model.Version) ([]task.Task, map[string]map[string]string, error)
}
TaskPrioritizer is responsible for taking in a slice of tasks, and ordering them according to which should be run first.
type Unit ¶
type Unit struct {
// contains filtered or unexported fields
}
Unit is a holder of a group of related tasks which should be scheculded together. Typically these represent task groups, tasks, and their dependencies, or even all tasks of a version. All tasks in a Unit must be unique with regards to their ID.
func MakeUnit ¶
MakeuUnit constructs a new unit, caching a reference to the distro in the unit. It's valid to pass a nil here.
func (*Unit) Export ¶
Export returns an unordered sequence of tasks from unit. All tasks are unique.
func (*Unit) RankValue ¶
RankValue returns a point value for the tasks in the unit that can be used to compare units with each other.
Generally, higher point values are given to larger units and for units that have been in the queue for longer, with longer expected runtimes. The tasks' priority acts as a multiplying factor.
type UnitCache ¶
UnitCache stores an unordered collection of schedulable units. The Unit type holds one or more tasks, but is handled by the scheduler as a single object. While the constituent tasks in a unit have an order, the unit themselves are an intermediate abstraction for the planner which represent task groups, tasks with their dependencies, or the tasks from a single version
func (UnitCache) AddNew ¶
AddNew adds an entire unit to a cache with the specified ID. If the cached item exists, AddNew extends the existing unit with the tasks from the passed unit.
func (UnitCache) Create ¶
Create makes a new unit around the existing task, caching it with the specified key, and returning the resulting unit. If there is an existing cache item with the specified ID, then Create extends that unit with this task. In both cases, the resulting unit is returned to the caller.
Source Files