spec

Documentation

Overview

Package spec is the job definition: YAML in, canonical JSON out.

Why there is an IR at all

The engine never reads YAML. It reads a canonical JSON document called paceq.job.v1, carrying a spec_hash, and this package is the only thing that produces one (03 section 3.2, SYNTESE section 3.3). That buys three things for the price of one encoder. A second frontend is a second frontend, not a second engine. Versioning a job is comparing two hashes rather than diffing two files. And "show me exactly what ran" stays true after the file on disk has been edited, because the run stored the hash of what it ran.

The rule that makes the hash worth anything is that defaults are materialised before hashing. A job that writes max_concurrent: 1 and a job that leaves it out are the same job, and they must hash the same, or every apply after a tidy-up of the file would record a version that changed nothing.

Why the decoder is written out by hand

Parse walks the YAML syntax tree itself rather than unmarshalling into the structs. Three things fall out of that which reflection does not give:

• Every diagnostic carries the line and column of the thing it is about, which is what makes the excerpt and the caret possible.
• An unknown field is caught with the name the user typed still in hand, so the message can say what they probably meant.
• Scalars are read as the text that is in the file. YAML 1.1 turns the country code NO into false, and a job that inherits an environment variable called NO deserves better than a silent boolean.

Limits before decoding

A YAML file that expands to more than it says is the reason for every limit in this package (08 T13). The syntax tree is bounded work: an alias in it is a name, not a copy. So the file size, the nesting depth, the alias count and the node count are all checked on the tree, before anything is expanded, and the decoder that does expand aliases carries a budget it cannot exceed. A billion laughs file is refused having allocated a syntax tree the size of the file it came from.

What is deliberately not here

No templating, in any form (SYNTESE section 3.3). No needs semantics: the field is parsed and carried into the IR, and the cycle detection that gives it meaning is M4-01. Schedules and sensors parse and validate but activate in M2 and M3. Nothing in this package reads a database or starts a process.

Index

• Constants
• Variables
• func Canonical(j *Job) []byte
• func CheckGlobalSensorNames(jobs []NamedJob) diag.List
• func Codes() []string
• func Collect(paths []string) ([]string, error)
• func FormatDuration(d time.Duration) string
• func Hash(canonical []byte) string
• func IsJobFile(name string) bool
• func LoadFile(path string) (*Job, Source, diag.List)
• func ParseDuration(text string) (time.Duration, error)
• type Hashed
  • func Compile(j *Job) Hashed
• type Job
  • func FromIR(data []byte) (*Job, error)
  • func Parse(path string, src []byte) (*Job, diag.List)
• type NamedJob
• type Retry
• type Schedule
• type Sensor
• type Source
  • func ReadFile(path string) (Source, diag.List)
• type Step

Constants

const (
	// CodeSyntax is a file the YAML parser cannot read at all.
	CodeSyntax = "PQ1000"
	// CodeFileTooLarge is a job file over MaxFileBytes, refused before it is
	// decoded.
	CodeFileTooLarge = "PQ1001"
	// CodeBadName is a missing name, or one that does not match NamePattern.
	CodeBadName = "PQ1002"
	// CodeMissingField is a required field that is not there, or is empty.
	CodeMissingField = "PQ1003"
	// CodeBadValue is a field with the wrong type or a value outside what it
	// accepts.
	CodeBadValue = "PQ1004"
	// CodeTooDeep is nesting past MaxDepth.
	CodeTooDeep = "PQ1005"
	// CodeTooManyAliases is more than MaxAliases aliases, or an alias that
	// refers to itself.
	CodeTooManyAliases = "PQ1006"
	// CodeTooLarge is a file with more nodes than MaxNodes, or one that
	// expands past MaxExpandedNodes.
	CodeTooLarge = "PQ1008"
	// CodeTooManySteps is more than MaxSteps steps.
	CodeTooManySteps = "PQ1009"
	// CodeRunNotAList is run written as a string.
	CodeRunNotAList = "PQ1010"
	// CodeRunEmpty is run written as an empty list.
	CodeRunEmpty = "PQ1011"
	// CodeRunNotAbsolute is a run[0] that is not an absolute path, with no
	// shell to look it up.
	CodeRunNotAbsolute = "PQ1012"
	// CodeBadDuration is a duration paceq does not read.
	CodeBadDuration = "PQ1020"
	// CodeTimeoutTooLong is a timeout over MaxJobTimeout.
	CodeTimeoutTooLong = "PQ1021"
	// CodeBadConcurrency is max_concurrent below one.
	CodeBadConcurrency = "PQ1030"
	// CodeUnknownField is a field name paceq does not know, with the nearest
	// one it does.
	CodeUnknownField = "PQ1040"
	// CodeDuplicateKey is the same key twice in one mapping.
	CodeDuplicateKey = "PQ1041"
	// CodeMergeKey is a << merge key, which paceq does not resolve.
	CodeMergeKey = "PQ1042"
	// CodeTag is an explicit YAML tag, which paceq does not honour.
	CodeTag = "PQ1043"
	// CodeTooManyProblems is where paceq stopped reading a file that had more
	// wrong with it than a report can carry.
	CodeTooManyProblems = "PQ1044"
	// CodeDuplicateStep is two steps with the same name.
	CodeDuplicateStep = "PQ2001"
	// CodeUnknownNeed is a needs entry naming a step that does not exist.
	CodeUnknownNeed = "PQ2002"
	// CodeUnknownTimezone is a schedule zone the time zone database does not
	// have.
	CodeUnknownTimezone = "PQ2010"
	// CodeShell is the warning that a step's command reaches a shell.
	CodeShell = "W1001"
	// CodeInheritEnv is the warning that a job takes variables from the
	// environment paceq itself was started in.
	CodeInheritEnv = "W1002"

	// CodeSensorBadName is a sensor name that is missing or does not match
	// NamePattern.
	CodeSensorBadName = "PQ4101"
	// CodeSensorNameTaken is a sensor name another sensor in the job, or a
	// sensor in another job, already carries.
	CodeSensorNameTaken = "PQ4102"
	// CodeSensorKind is a kind other than exec: in 1.0 the built in sensor
	// types arrive in v0.3, not earlier.
	CodeSensorKind = "PQ4103"
	// CodeSensorRun is run written as a string, as an empty list, or with an
	// empty argument in it.
	CodeSensorRun = "PQ4104"
	// CodeSensorIntervalMin is an interval under the one second floor.
	CodeSensorIntervalMin = "PQ4105"
	// CodeSensorMinInterval is a min_interval over the interval itself.
	CodeSensorMinInterval = "PQ4106"
	// CodeSensorTimeout is a timeout outside the [1s, 5m] range.
	CodeSensorTimeout = "PQ4107"
	// CodeSensorTriggers is a max_triggers_per_tick outside [1, 10000].
	CodeSensorTriggers = "PQ4108"
	// CodeSensorWorkdir is the warning that a workdir does not exist, or is
	// not an absolute path.
	CodeSensorWorkdir = "PQ4109"
	// CodeSensorEnvKey is an env key with the reserved PULSEQ_ prefix.
	CodeSensorEnvKey = "PQ4110"
)

The diagnostic codes this package raises. The series are fixed by 03 section 8.1: PQ1xxx is parsing and schema, PQ2xxx is semantics, W1xxx is a warning.

A code is a public interface the moment it is printed: it goes into scripts that grep for it and into `paceq error PQ1040`. Codes are therefore added, never reused for something else.

const (
	// MaxFileBytes is one job file. A job definition that needs a megabyte is
	// carrying data that belongs somewhere else.
	MaxFileBytes = 1 << 20
	// MaxDepth is how deeply the file may nest. The schema itself never goes
	// past six.
	MaxDepth = 32
	// MaxAliases is how many aliases one file may use. Anchors are useful for
	// a shared env block; a hundred of them is a program.
	MaxAliases = 100
	// MaxNodes is the size of the syntax tree, before any alias is resolved.
	MaxNodes = 20000
	// MaxSteps is the same ceiling M4-01 puts on a DAG.
	MaxSteps = 200
	// MaxExpandedNodes is the decoder's budget. Every node it visits costs
	// one, including each visit through an alias, so nested aliases cannot
	// multiply their way past the tree limits above.
	MaxExpandedNodes = 200000
)

Limits are the parser's refusals, all of them checked before the file is expanded (08 T13). They are exported because the messages quote them and the tests assert against the same numbers the code uses.

const (
	// DefaultTimeout is what a job without a timeout gets. A timeout is
	// mandatory (08 section 3.2); the default is what makes it mandatory
	// without making every file say so.
	DefaultTimeout = time.Hour
	// MaxJobTimeout is the system ceiling. A job that legitimately runs longer
	// than a day is a service, and paceq does not supervise services.
	MaxJobTimeout = 24 * time.Hour
	// DefaultMaxConcurrent is one. Cron has the opposite default, and the
	// flock wrappers people write around cron jobs are what it costs them
	// (09 section 7, US-02).
	DefaultMaxConcurrent = 1
	// DefaultTimezone is the zone a schedule without one runs in. UTC rather
	// than the daemon's local zone: a schedule that means something different
	// depending on which machine reads it is not explainable.
	DefaultTimezone = "UTC"
)

Defaults are materialised into the IR before it is hashed, so that leaving a field out and writing its default produce the same job and the same hash.

const (
	DefaultBackoff  = BackoffExponential
	DefaultInitial  = 30 * time.Second
	DefaultMaxDelay = 10 * time.Minute
	DefaultJitter   = JitterFull
)

Retry defaults, taken from the project defaults block in 03 section 3.4 so the two cannot disagree.

const (
	BackoffExponential = "exponential"
	BackoffFixed       = "fixed"
	JitterFull         = "full"
	JitterNone         = "none"
)

The values the enumerated fields accept.

const (
	// SensorIntervalMin is the lowest an interval may go. Sensorers are
	// evaluated by one shared runtime; sub-second polling is a non-goal
	// (SCOPE.md section 3.19).
	SensorIntervalMin = time.Second

	// DefaultSensorTimeout is what a sensor without one gets. The same
	// default the design chose (03 section 4.5, 02 section 5.5).
	DefaultSensorTimeout = 30 * time.Second

	// SensorTimeoutMin is the floor every sensor timeout must clear.
	SensorTimeoutMin = time.Second

	// SensorTimeoutMax is the system ceiling. A sensor that needs more than
	// this should chunk its own work via max_triggers_per_tick, not ask the
	// runtime to wait on it (M1-06, the step ceiling, is the same rule).
	SensorTimeoutMax = 5 * time.Minute

	// DefaultSensorMaxTriggers is the default ceiling on one tick.
	DefaultSensorMaxTriggers = 100

	// SensorMaxTriggersHi is what max_triggers_per_tick may be raised to.
	SensorMaxTriggersHi = 10_000
)

The sensor bounds, all of them public because the messages quote them and the tests assert against the same numbers the code uses.

const (
	// OverlapSkip stands a tick down when the concurrency limit is held. It
	// is the default, because the user paceq replaces is flock -n, which
	// skips too: the least surprising behaviour, except the stand-down is
	// recorded with its reason instead of vanishing.
	OverlapSkip = "skip"

	// OverlapQueue materialises the run anyway, deferred: queued with
	// available_at in the future and defer_reason set, so it starts when a
	// slot frees instead of being dropped.
	OverlapQueue = "queue"
)

const DefaultOverlap = OverlapSkip

DefaultOverlap is OverlapSkip. Like DefaultMaxConcurrent this is a product decision, not a technical one, and it is spelled out here so both ends of the decode read it from one place.

const DefaultSensorKind = "exec"

The default kind is the only kind. One adapter, a subprocess that writes JSON to stdout (10 section 5, F4b): a second kind is a migration and a new evaluator, never a flag somebody sets by accident. The value is spelled out here so every reader of what sensors accept takes it from one place.

const MaxFlowMarkers = 2000

MaxFlowMarkers bounds the [ and { characters in a file, counted on the raw bytes before the YAML parser runs.

The reason is the parser's cost, not the schema's shape. Parsing a flow collection is quadratic in how deeply it nests, so a one line file of two hundred thousand opening brackets keeps a core busy for minutes. The exact depth limit below is checked on the syntax tree, which is the right place for it, but that check only runs once there is a tree. This one is what makes sure there is one.

The count is taken without knowing which brackets sit inside a quoted string, so a file could in principle be refused for brackets that are only text. It would need two thousand of them, in a file that MaxNodes refuses anyway.

const NamePattern = "^[a-z0-9][a-z0-9_-]{0,63}$"

NamePattern is the rule as a message can quote it.

const SchemaName = "paceq.job.v1"

SchemaName is what the canonical JSON calls itself. It is the first thing a consumer reads and the thing that makes a second version of the IR possible without guessing.

Variables

var ErrDuration = errors.New("not a duration paceq reads")

ErrDuration is every way a duration can be unreadable. The message says which one; callers only need to know that the text was not a duration.

var Extensions = []string{".yaml", ".yml"}

Extensions are the file names a directory walk picks up as job files.

Functions

func Canonical

func Canonical(j *Job) []byte

Canonical is the job as the engine reads it: paceq.job.v1, with every default materialised and every key in a fixed place. Nothing about the result depends on map iteration order, on the order fields were written in the YAML file, or on the Go release that built the binary.

func CheckGlobalSensorNames

func CheckGlobalSensorNames(jobs []NamedJob) diag.List

CheckGlobalSensorNames rejects two jobs that define the same sensor name. A sensor name is the sensor row's primary key across every job, so two owners of one name cannot both materialise; the later apply would silently steal the row from the first. Each diagnostic points at one conflicting job and names the file that already owns the name.

func Codes

func Codes() []string

Codes is every code this package can raise, in the order they are declared. The catalogue behind `paceq error` is checked against it, and a test reads the package source to prove nothing raises a code that is missing from here.

func Collect

func Collect(paths []string) ([]string, error)

Collect turns what was typed on a command line into the list of job files to read. A path that names a file is taken as it is, whatever it is called; a path that names a directory is walked for the job file extensions.

The result is sorted, so two runs over the same tree report in the same order and a golden test can compare them.

func FormatDuration

func FormatDuration(d time.Duration) string

FormatDuration writes a duration the way a job file would, so a message that quotes a ceiling reads like the field it is about.

func Hash

func Hash(canonical []byte) string

Hash is the spec_hash: sha256 over the canonical document, prefixed with the algorithm so a stored hash says what produced it.

func IsJobFile

func IsJobFile(name string) bool

IsJobFile reports whether a directory walk picks a name up.

func LoadFile

func LoadFile(path string) (*Job, Source, diag.List)

LoadFile reads and parses one job file.

func ParseDuration

func ParseDuration(text string) (time.Duration, error)

ParseDuration reads a duration the way a job file writes one: a run of number and unit pairs, such as 30s, 45m, 1h30m or 1500ms. The result is always a whole number of milliseconds, which is the unit the IR stores, so a value that cannot be one is refused rather than silently rounded.

Negative and zero durations are refused. Every field that takes a duration is a timeout or a delay, and neither has a meaning at zero.

Types

type Hashed

type Hashed struct {
	Job       *Job
	Canonical []byte
	Hash      string
}

Hashed is a job and the exact bytes its hash was taken over. The two travel together because a hash without the document it covers cannot be checked.

func Compile

func Compile(j *Job) Hashed

Compile is the whole pipeline over a job that has already parsed: canonical bytes and the hash over them.

type Job

type Job struct {
	Name          string
	Description   string
	Env           map[string]string
	EnvFile       string
	InheritEnv    []string
	Workdir       string
	Timeout       time.Duration
	MaxConcurrent int
	Steps         []Step
	Schedules     []Schedule
	Sensors       []Sensor
}

Job is one job definition, after parsing and with every default materialised. It is the engine facing type: no positions, no source, nothing that only makes sense while a file is open.

func FromIR

func FromIR(data []byte) (*Job, error)

FromIR reads a canonical document back into a Job. The engine materialises every run from these bytes, which are frozen in job_versions.spec_json, so this function is what makes a run independent of the file it was applied from: the version row is immutable, and reading it back through here yields exactly the steps, timeouts and edges the hash was taken over.

The decoder is strict about shape rather than permissive: a key the v1 writer never emits is a corrupted or future document, and running half of one would be worse than refusing it. Every refusal names the key path.

func Parse

func Parse(path string, src []byte) (*Job, diag.List)

Parse reads one job file. The bytes are the file; path is only ever used to name it in a diagnostic, so a caller with the content in hand can parse without touching a disk.

A job comes back only when nothing refused it. Warnings come back with the job, because a job with a warning is a job paceq will run.

type NamedJob

type NamedJob struct {
	Path string
	Job  *Job
}

NamedJob is one parsed job and the path it came from, the input the checks that need the whole catalog run over.

type Retry

type Retry struct {
	Max      int
	Backoff  string
	Initial  time.Duration
	MaxDelay time.Duration
	Jitter   string
}

Retry is what happens after a step fails.

type Schedule

type Schedule struct {
	Name     string
	Cron     string
	Timezone string
	Overlap  string
}

Schedule is a cron expression and the zone it is read in. It parses and validates here and activates in M2: the expression itself is checked by the cron parser that arrives with the scheduler. Overlap says what a firing does when the job's max_concurrent is already held: "skip" (the default) stands down, "queue" defers the run into the future.

type Sensor

type Sensor struct {
	// Name is unique within the job and across every job in the catalog,
	// because it is the sensor row's primary key.
	Name string

	// Kind is always "exec" in 1.0. The field is read so a later kind is a
	// validation error with a message that says which release brings it,
	// rather than a value that silently means nothing.
	Kind string

	// Run is argv. There is no string form, so nothing here is ever split,
	// quoted or expanded by a shell (08 section 3.2).
	Run []string

	// Workdir is the directory the sensor process starts in. Empty means the
	// engine's own working directory.
	Workdir string

	// Env is the environment the sensor process is started with, in addition
	// to (never replacing) the job's own inherited baseline.
	Env map[string]string

	// Interval is how often the sensor is evaluated. Minimum one second.
	Interval time.Duration

	// MinInterval is the absolute lower bound between two starts of the same
	// sensor, even when a retry makes an immediate re-evaluation legal. It
	// defaults to one second.
	MinInterval time.Duration

	// Timeout is the ceiling on one evaluation. Defaults to 30s; the hard
	// ceiling is SensorTimeoutMax.
	Timeout time.Duration

	// MaxTriggersPerTick is how many triggers one evaluation may admit. It is
	// the chunking knob that keeps a burst from flooding the queue.
	MaxTriggersPerTick int

	// Paused is the initial state on first materialisation. Pausing a sensor
	// is an operator decision and survives re-apply.
	Paused bool

	// Description is what the sensor is for, for the people who read the job.
	Description string
}

Sensor is an external trigger. It parses and validates here and materialises into the sensors table at apply (M3-01). The type names and their own fields land here too; in 1.0 there is exactly one kind, exec.

type Source

type Source struct {
	Path  string
	Bytes []byte
}

Source is one job file as it was read, kept so a diagnostic can be rendered with the excerpt it points at after the file is closed.

func ReadFile

func ReadFile(path string) (Source, diag.List)

ReadFile reads one job file without ever holding more than the limit in memory. That is what makes the size refusal real: a two megabyte file is refused having read one megabyte and one byte of it, and the YAML parser never sees it.

type Step

type Step struct {
	Name string
	// Run is argv. There is no string form, so nothing here is ever split,
	// quoted or expanded by a shell (08 section 3.2).
	Run []string
	// Shell is the explicit opt in that hands Run to a shell. It carries a
	// validation warning wherever it is true.
	Shell   bool
	Workdir string
	// Timeout is this step's own ceiling. Zero means the step is bounded by
	// the job's timeout rather than by one of its own.
	Timeout time.Duration
	Retry   *Retry
	// Needs is carried into the IR and means nothing yet. The cycle detection
	// and the topological order are M4-01; in M1 the steps run in file order.
	Needs []string
}

Step is one command in a job.