metrics

package
v0.1.13-1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 27, 2023 License: MIT Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source 
const (

	// Metric Family Types (these likely can be changed)
	SystemFamily          = "system"
	StorageFamily         = "storage"
	MachineLearningFamily = "machine-learning"
	NetworkFamily         = "network"
	SimulationFamily      = "simulation"
	SolverFamily          = "solver"

	// Generic (more than one type, CPU/io, etc)
	ProxyAppFamily    = "proxyapp"
	PerformanceFamily = "performance"
)

Variables

View Source 
var (
	DefaultEntrypointScript = "/metrics_operator/entrypoint-0.sh"
)

These are common templates for application metrics

View Source 
var (
	Registry = map[string]Metric{}
)
View Source 
var (
	RegistrySet = make(map[string]MetricSet)
)
View Source 
var (
	// Keep this short so DNS doesn't risk overflow
	// This is the default Replicated Job Name optional for use
	ReplicatedJobName = "m"
)
View Source 
var (

	// TemplateConvertHostnames assumes a hostnames.txt to write to hostlist.txt
	TemplateConvertHostnames = `` /* 447-byte string literal not displayed */

)

Functions

func AssembleReplicatedJob 

func AssembleReplicatedJob(
	set *api.MetricSet,
	shareProcessNamespace bool,
	pods int32,
	completions int32,
	jobname string,
	soleTenancy bool,
) (*jobset.ReplicatedJob, error)

AssembleReplicatedJob is used by metrics to assemble a custom, replicated job.

func GetJobSet 

func GetJobSet(
	spec *api.MetricSet,
	set *MetricSet,
) (*jobset.JobSet, []*specs.ContainerSpec, error)

GetJobSet is called by the controller to return a JobSet for the MetricSet

func Metadata 

func Metadata(set *api.MetricSet, metric *Metric) string

Default metadata (in JSON) to also put at the top of logs for parsing I'd like to improve upon this manual approach, it's a bit messy.

func Register 

func Register(m Metric)

Register a new Metric type, adding it to the Registry

Types

type BaseMetric 

type BaseMetric struct {
	Identifier string
	Summary    string
	Container  string
	Workdir    string

	// A custom container can be used to replace the application
	// (typically advanced users only)
	CustomContainer string
	ResourceSpec    *api.ContainerResources
	AttributeSpec   *api.ContainerSpec

	// If we ask for sole tenancy, we assign 1 pod / hostname
	SoleTenancy bool

	// A metric can have one or more addons
	Addons map[string]*addons.Addon
}

BaseMetric provides shared attributes across Metric types

func (BaseMetric) AddAddons 

func (m BaseMetric) AddAddons(
	spec *api.MetricSet,
	rjs []*jobset.ReplicatedJob,

	containerSpecs []*specs.ContainerSpec,
) ([]*specs.ContainerSpec, error)

Add registered addons to replicated jobs Container specs returned are assumed to be config maps that need to be written

func (BaseMetric) Attributes 

func (m BaseMetric) Attributes() *api.ContainerSpec

func (BaseMetric) Description 

func (m BaseMetric) Description() string

Description returns the metric description

func (BaseMetric) GetAddons 

func (m BaseMetric) GetAddons() []*addons.Addon

Addons returns a list of addons, removing them from the key value lookup

func (BaseMetric) HasSoleTenancy 

func (m BaseMetric) HasSoleTenancy() bool

func (*BaseMetric) Image 

func (m *BaseMetric) Image() string

Container

func (*BaseMetric) InitAddons 

func (m *BaseMetric) InitAddons()

InitAddons ensures we don't have an empty map

func (BaseMetric) ListOptions 

func (m BaseMetric) ListOptions() map[string][]intstr.IntOrString

func (BaseMetric) Name 

func (m BaseMetric) Name() string

Name returns the metric name

func (*BaseMetric) RegisterAddon 

func (m *BaseMetric) RegisterAddon(addon *addons.Addon)

RegisterAddon adds an addon to the set, assuming it's already validated

func (*BaseMetric) ReplicatedJobs 

func (m *BaseMetric) ReplicatedJobs(spec *api.MetricSet) ([]*jobset.ReplicatedJob, error)

Default replicated jobs will generate for N pods, with no shared process namespace (e.g., storage)

func (BaseMetric) Resources 

func (m BaseMetric) Resources() *api.ContainerResources

Return container resources for the metric container

func (*BaseMetric) SetContainer 

func (m *BaseMetric) SetContainer(container string)

Set a custom container

func (BaseMetric) SetDefaultOptions 

func (m BaseMetric) SetDefaultOptions(metric *api.Metric)

SetDefaultOptions that are shared (possibly)

func (BaseMetric) SuccessJobs 

func (m BaseMetric) SuccessJobs() []string

Jobs required for success condition (n is the netmark run)

func (BaseMetric) Validate 

func (m BaseMetric) Validate(set *api.MetricSet) bool

Validation

type LauncherWorker 

type LauncherWorker struct {
	BaseMetric
	ResourceSpec  *api.ContainerResources
	AttributeSpec *api.ContainerSpec

	// A metric can have one or more addons
	Addons []*api.MetricAddon

	// Most laucher workers have a command
	Command string
	Prefix  string

	// Scripts
	WorkerScript      string
	LauncherScript    string
	LauncherLetter    string
	WorkerContainer   string
	LauncherContainer string
	WorkerLetter      string
}

LauncherWorker is a launcher + worker setup for apps. These need to be accessible by other packages (and not conflict with function names)

func (*LauncherWorker) AddWorkers 

func (m *LauncherWorker) AddWorkers(spec *api.MetricSet) (*jobset.ReplicatedJob, error)

AddWorkers generates worker jobs, only if we have them

func (LauncherWorker) Family 

func (m LauncherWorker) Family() string

Family returns a generic performance family

func (*LauncherWorker) GetCommonPrefix 

func (m *LauncherWorker) GetCommonPrefix(
	meta string,
	command string,
	hosts string,
) string

GetCommonPrefix returns a common prefix for the worker/ launcher script, setting up hosts, etc.

func (*LauncherWorker) GetHostlist 

func (m *LauncherWorker) GetHostlist(spec *api.MetricSet) string

Get common hostlist for launcher/worker app

func (*LauncherWorker) GetLauncherContainerSpec 

func (m *LauncherWorker) GetLauncherContainerSpec(
	entrypoint specs.EntrypointScript,
) specs.ContainerSpec

func (*LauncherWorker) GetWorkerContainerSpec 

func (m *LauncherWorker) GetWorkerContainerSpec(
	entrypoint specs.EntrypointScript,
) specs.ContainerSpec

func (*LauncherWorker) PrepareContainers 

func (m *LauncherWorker) PrepareContainers(
	spec *api.MetricSet,
	metric *Metric,
) []*specs.ContainerSpec

func (*LauncherWorker) ReplicatedJobs 

func (m *LauncherWorker) ReplicatedJobs(spec *api.MetricSet) ([]*jobset.ReplicatedJob, error)

Replicated Jobs are custom for a launcher worker

func (*LauncherWorker) SetDefaultOptions 

func (m *LauncherWorker) SetDefaultOptions(metric *api.Metric)

Set default options / attributes for the launcher metric

func (*LauncherWorker) SuccessJobs 

func (m *LauncherWorker) SuccessJobs() []string

Jobs required for success condition (n is the LauncherWorker run)

func (LauncherWorker) Validate 

func (m LauncherWorker) Validate(spec *api.MetricSet) bool

Validate that we can run a network. At least one launcher and worker is required

type Metric 

type Metric interface {

	// Metadata
	Name() string
	Description() string
	Family() string
	Url() string

	// Container attributes
	Image() string
	SetContainer(string)

	// Options and exportable attributes
	SetOptions(*api.Metric)
	Options() map[string]intstr.IntOrString
	ListOptions() map[string][]intstr.IntOrString

	// Validation and append addons
	Validate(*api.MetricSet) bool
	RegisterAddon(*addons.Addon)
	AddAddons(*api.MetricSet, []*jobset.ReplicatedJob, []*specs.ContainerSpec) ([]*specs.ContainerSpec, error)
	GetAddons() []*addons.Addon

	// Attributes for JobSet, etc.
	HasSoleTenancy() bool
	ReplicatedJobs(*api.MetricSet) ([]*jobset.ReplicatedJob, error)
	SuccessJobs() []string
	Resources() *api.ContainerResources
	Attributes() *api.ContainerSpec

	// Prepare Containers. These are used to generate configmaps,
	// and populate the respective replicated jobs with containers!
	PrepareContainers(*api.MetricSet, *Metric) []*specs.ContainerSpec
}

A general metric is a container added to a JobSet

func GetMetric 

func GetMetric(metric *api.Metric, set *api.MetricSet) (Metric, error)

GetMetric returns a metric, if it is known to the metrics operator We also confirm that the addon exists, validate, and instantiate it.

type MetricSet 

type MetricSet struct {
	// contains filtered or unexported fields
}

A MetricSet includes one or more metrics that are assembled into a JobSet

func (*MetricSet) Add 

func (ms *MetricSet) Add(metric *Metric)

func (MetricSet) Exists 

func (m MetricSet) Exists(metric *Metric) bool

func (MetricSet) HasSoleTenancy 

func (m MetricSet) HasSoleTenancy() bool

Determine if any metrics in the set need sole tenancy This is defined on the level of the jobset for now

func (MetricSet) Metrics 

func (m MetricSet) Metrics() []*Metric

type SingleApplication 

type SingleApplication struct {
	BaseMetric
}

SingleApplication is a Metric base for a simple application metric be accessible by other packages (and not conflict with function names)

func (*SingleApplication) ApplicationContainerSpec 

func (m *SingleApplication) ApplicationContainerSpec(
	preBlock string,
	command string,
	postBlock string,
) []*specs.ContainerSpec

func (SingleApplication) Family 

func (m SingleApplication) Family() string

Default SingleApplication is generic performance family

func (SingleApplication) HasSoleTenancy 

func (m SingleApplication) HasSoleTenancy() bool

func (SingleApplication) Options 

func (m SingleApplication) Options() map[string]intstr.IntOrString

func (*SingleApplication) ReplicatedJobs 

func (m *SingleApplication) ReplicatedJobs(spec *api.MetricSet) ([]*jobset.ReplicatedJob, error)

Replicated Jobs are custom for a launcher worker

type StorageGeneric 

type StorageGeneric struct {
	BaseMetric
}

func (StorageGeneric) Family 

func (m StorageGeneric) Family() string

Family returns the storage family

func (StorageGeneric) HasSoleTenancy 

func (m StorageGeneric) HasSoleTenancy() bool

By default assume storage does not have sole tenancy

func (*StorageGeneric) StorageContainerSpec 

func (m *StorageGeneric) StorageContainerSpec(
	preBlock string,
	command string,
	postBlock string,
) []*specs.ContainerSpec

StorageContainerSpec gets the storage container spec This is identical to the application spec and could be combined

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.