Documentation
¶
Overview ¶
Package runner executes a single step as an operating system process, owning process groups, signals and log pipes.
The runner is where paceq actually does something: it starts a user command, holds it on a short leash and reports the outcome precisely. Three ground rules shape everything here.
The orchestrator process never runs user code (plan 04, ground rule 1). Everything user defined is a subprocess with a hard timeout and its own process group. Run is synchronous: one call, one attempt, one verdict.
Setpgid is not optional (plan 05, section 6.5). Without its own process group a job's grandchildren outlive the kill: the shell starts python, python starts curl, and the curl survives as a zombie holding files and ports. Every kill in this package addresses the negative process group id, never a bare pid.
The environment is deny by default (plan 08, section 3.2). A job does not inherit the daemon's environment. "Works in the shell, fails in cron" is the number one cron trap, and it is solved by making the environment explicit instead of inheriting more.
The frozen environment contract ¶
The child environment is built in layers; a key set by a later layer replaces the same key from an earlier one. The PACEQ_ prefix is reserved: no job layer may set, inherit or read from a file a key with that prefix.
Layer 1, baseline (deny by default):
PATH fixed default, never inherited from the daemon
HOME taken from the runner process if set
TZ taken from the runner process if set
LANG taken from the runner process if set
Layer 2, the context contract, set on every run:
PACEQ_RUN_ID the run's ULID
PACEQ_JOB job name
PACEQ_STEP step name
PACEQ_ATTEMPT 1-based attempt number
PACEQ_RUN_KEY dedup key, empty when the run had none
PACEQ_IDEMPOTENCY_KEY sha256(run_id + ":" + step_name), first 32 hex
characters; stable across retries and duplicates
of the same step in the same run, so a user step
can use it as an idempotency key downstream
PACEQ_SCHEDULED_FOR RFC3339 UTC, empty for manual runs
PACEQ_PARAMS the step's params as a JSON object
PACEQ_OUTPUT path of the NDJSON file the step may write
artifacts and params to; created by the runner
before the command starts
PACEQ_INPUTS merged JSON from upstream steps, "{}" in M1
Layer 3, InheritEnv: only the named variables, copied from the runner
process when they exist there.
Layer 4, EnvFile: a KEY=VALUE file that must have mode 0600 exactly; a
looser mode is refused (fail closed).
Layer 5, Env: the job's own environment, the most specific and most
reviewed layer, wins over every other user layer.
Outcomes ¶
Run reports one of five outcomes with a reason code each, so explain can tell "command not found" from "exit 1" from "killed by a signal" from "hit the deadline": three different incidents, three different fixes. The table is executable in result_test.go.
Succeeded exit 0 STEP_SUCCEEDED
Failed exit N STEP_FAILED_NONZERO_EXIT
with transient=true when N=75, the EX_TEMPFAIL convention
Signalled killed by a signal STEP_FAILED_SIGNAL
exit code reported as 128+signal
TimedOut deadline hit, group killed STEP_FAILED_TIMEOUT
SpawnFailed the process never started STEP_FAILED_SPAWN
(missing binary, no execute permission, missing workdir,
refused path, unreadable env file); no command side effects,
so a retry is always safe
A clean exit status is direct evidence of completion and wins over a deadline that fired in the same instant; a signal death under an active deadline is the runner's own kill and reports as TimedOut.
Time ¶
All measured time comes from the clock.Clock in the Spec. The one context handed to os/exec carries cancellation plumbing only; the deadline itself is a clock timer, so a fake clock drives every timing decision in tests.
Platform ¶
Process groups and the core limit are unix features. On other platforms the runner still works but degrades to pid targeted kills and no rlimit change; the supported targets (linux, darwin) all take the unix path.
Index ¶
Constants ¶
const ( ReasonSucceeded = "STEP_SUCCEEDED" ReasonNonzeroExit = "STEP_FAILED_NONZERO_EXIT" ReasonTimeout = "STEP_FAILED_TIMEOUT" ReasonSpawn = "STEP_FAILED_SPAWN" ReasonSignal = "STEP_FAILED_SIGNAL" )
The reason codes this package emits. They follow the event catalogue in the observability plan; when the reason package lands (M1-05) these constants become its values without a contract change.
const ( // DefaultTimeout is the timeout a job gets when no layer above the runner // set one. A job without a deadline is a job that hangs forever one day. DefaultTimeout = time.Hour // MaxTimeout is the hard system cap. The validator refuses anything // larger; the runner repeats the check as a second line of defence. MaxTimeout = DefaultTimeout // DefaultKillGrace is how long a job may keep running after SIGTERM // before SIGKILL ends the whole group. DefaultKillGrace = 10 * time.Second )
const DefaultPath = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
DefaultPath is the fixed PATH every job starts from. It is never inherited: a daemon started from a developer shell must not hand its search path to a production job.
Variables ¶
var ErrInvalidSpec = errors.New("invalid runner spec")
ErrInvalidSpec wraps every refusal Run makes before touching a process. These are caller bugs: no attempt was made, no attempt is retryable, and the fix is in the spec, not the job.
Functions ¶
func ExitStatus ¶
func ExitStatus(st *os.ProcessState) (code int, signal string, signalled bool)
ExitStatus is the public reading of a finished process's wait status: the exit code for a normal exit, or the canonical signal name for a signal death. The sensor evaluator reads the same facts through this seam instead of re-deriving the platform split the classifier above keeps private.
func KillAllProcessGroups ¶
KillAllProcessGroups signals every registered group. Delivery failures are ignored on purpose: ESRCH only means the group is already gone, which is the outcome a hard stop wants anyway. It is called from the second-signal path immediately before exit, so nothing here may block.
func RegisterProcessGroup ¶
func RegisterProcessGroup(pgid int) func()
RegisterProcessGroup is the public form of registerGroup. The sensor evaluator calls it so a daemon hard stop (the second signal, which sweeps every group through KillAllProcessGroups) reaches a sensor subprocess with the same certainty it reaches a step.
func SysProcAttr ¶
func SysProcAttr() *syscall.SysProcAttr
SysProcAttr is the public form of sysProcAttr. The sensor evaluator uses it so its subprocess gets the same process group guarantee as a step: the child leads its own group, and every escalation on it reaches the whole group.
Types ¶
type Escalator ¶
type Escalator struct {
// contains filtered or unexported fields
}
Escalator is the public face of the SIGTERM then SIGKILL sequence against one process group. The sensor evaluator builds on it directly instead of maintaining a second implementation of the grace escalation; the daemon's hard stop reaches its groups because Start registers them through RegisterProcessGroup.
func NewEscalator ¶
NewEscalator prepares the sequence against one future process group. grace is the tolerated silence between SIGTERM and SIGKILL.
func (*Escalator) Fire ¶
Fire sends SIGTERM to the whole group now and arms SIGKILL for it once the grace elapses. It is idempotent and returns the error exec's Cancel hook expects.
type Outcome ¶
type Outcome int
Outcome is the five value verdict a run can have. The taxonomy is a contract: explain and retry logic read it, so two different incidents must never share a value. The executable table lives in result_test.go.
type Result ¶
type Result struct {
ExitCode int
Signal string // "SIGKILL" and friends, empty when the process exited on its own
StartedAt int64
FinishedAt int64
Pgid int
Outcome Outcome
ReasonCode string
ReasonData map[string]any
Err error // the operating system cause, set for SpawnFailed only
}
Result is the full verdict on one attempt. StartedAt and FinishedAt are unix milliseconds read from the Spec's clock.
func Run ¶
Run starts the command in its own process group and returns what happened.
The error result is reserved for spec refusals (ErrInvalidSpec): the process was never attempted. Every operating system level failure, including a missing binary, comes back as a SpawnFailed Result with a nil error, because those are job outcomes a caller can log, explain and retry.
type RunContext ¶
type RunContext struct {
RunID string
Job string
Step string
Attempt int
RunKey string
Params map[string]any
ScheduledFor time.Time // zero means a manual run
}
RunContext is everything the runner knows about why this process exists. It becomes the PACEQ_ environment contract.
type Spec ¶
type Spec struct {
Argv []string
Shell bool // explicit opt-in: wrap in /bin/sh -c
Workdir string // relative paths resolve inside the runner's working directory root
Env map[string]string // the job's own environment, highest user layer
EnvFile string // KEY=VALUE file, mode 0600 exactly
InheritEnv []string // daemon variables to pass through, by name
Timeout time.Duration // mandatory, capped at MaxTimeout
KillGrace time.Duration // SIGTERM to SIGKILL gap; DefaultKillGrace when not positive
Ctx RunContext
Clock clock.Clock // nil means clock.System
Stdout io.Writer // nil means /dev/null
Stderr io.Writer // nil means /dev/null
OutputPath string // created before the command starts, handed over as PACEQ_OUTPUT
// OnStart fires once per successful spawn, before Run returns, with the
// child's pid. The engine persists a process baseline through it (issue
// #62): pid plus /proc start ticks on file at spawn time is what later
// lets the orphan sweep tell a surviving child of a dead executor from a
// recycled pid. It is not called for a refused or failed spawn, and a
// slow or failing callback delays the job's own bookkeeping by whatever
// it takes; callers keep it cheap and non-fatal.
OnStart func(pid int)
}
Spec is one process attempt. The zero value is not runnable: Argv, a positive Timeout within the cap and a Ctx are required.
Source Files