health

Documentation

Overview

Package health is used for quick measuring of cluster health. Health for a node is provided in a Stat struct, containing everything we want to know. A lot of the data is provided by the github.com/jaypipes/ghw.HostInfo struct.

To measure health requires invoking three different commands. We have found it convenient to structure this via three separate steps.

First is the command to list the nodes. Second is to create the struct with the command to invoke the scheduler to run a command on a node. Third is to run the command using the struct

These three steps result in running two commands: a lister and a gatherer.

The lister operation is based on the NodeList struct. The zero value of a NodeList is a command, arguments, and empty list. NewNodeList takes a command and arguments. Run runs the command and populates the list. The List is exported and can be filled other ways. The command and args can be as simple as 'echo', '1', '2', '3' if that is desired; the List will then be '1', '2', and '3'.

Gather runs a scheduler command to gather Stat structs. NewGather takes the scheduler command and arguments needed to *schedule* a process on a node. Run takes the command and arguments to be *run on each node*. Run produces a JSON-encoded []Stat.

Typical commands for NodeList: sinfo -N -h -o %N Simpler commands for NodeList: echo ...names Typical commands for Gather: srun -t 5 -i none -w Simpler commands for Gather: cat ...file

vitalsigns prints out vital statistics about a node as JSON. It currently uses only the jaypipes/ghw package, but more may come later.

vitalsigns prints out vital statistics about a node as JSON. It currently uses only the jaypipes/ghw package, but more may come later.

Index

• Variables
• type Gather
  • func (g *Gather) Run(cmd string, args ...string) ([]Stat, error)
• type GatheredStats
• type Kernel
• type NodeList
  • func NewNodeList(cmd string, args ...string) *NodeList
  • func (n *NodeList) NewGather(cmd string, args ...string) *Gather
  • func (n *NodeList) Run() error
• type Stat

Variables

var V = func(string, ...any) {}

Types

type Gather

type Gather struct {
	Nodes *NodeList
	// contains filtered or unexported fields
}

func (*Gather) Run

func (g *Gather) Run(cmd string, args ...string) ([]Stat, error)

Run gathers data from the cluster, running cmd and args on each node. How they are run on the node is determined by the cmd and args in the Gather struct. Almost every error that can occur will occur on a per-node basis; those errors are accumulated into the Stderr member of the Stat struct, not returned. The intent is that this will saved in storage, so including errors is important.

type GatheredStats

type GatheredStats struct {
	Stats []Stat
}

type Kernel

type Kernel struct {
	Version string `file:"/proc/version"`
	Modules string `file:"/proc/modules"`
	Drivers string `file:"/proc/devices"`
}

Kernel should hold all stats for the kernel. If needed, later, the tag can be a command as well. Since it is returned as JSON, it is no problem to change it over time.

type NodeList

type NodeList struct {
	List []string
	// contains filtered or unexported fields
}

func NewNodeList

func NewNodeList(cmd string, args ...string) *NodeList

NewNodeList creates a NodeList with an empty List. The List is populated by running cmd and args.

func (*NodeList) NewGather

func (n *NodeList) NewGather(cmd string, args ...string) *Gather

NewGather creates a Gather from a NodeList. The cmd and args are used to invoke a scheduler command (e.g. srun, sbatch) to invoke a command on each node in turn. The assumption is that the command pattern looks like cmd args... nodename command-for-node args-for-node It is possible that, over time, we will need something more flexible, but this works for most schedulers we know of.

func (*NodeList) Run

func (n *NodeList) Run() error

Run creates a list of nodes. The node names are assumed to be a list that can be split by strings.Fields.

type Stat

type Stat struct {
	Hostname string
	Info     *ghw.HostInfo
	Kernel   Kernel
	// This should be empty, but some packages behave badly.
	Stderr string
	Err    string
}