Documentation
¶
Overview ¶
Package server exposes the controller's HTTP API. Every route is plain HTTP so it passes through the tunnels that already carry ssh.
Index ¶
- Constants
- func WriteTTYFrame(w io.Writer, f TTYFrame) error
- type Assignment
- type Config
- type DescribeResponse
- type DeviceRecovery
- type DeviceSpec
- type DeviceView
- type Event
- type ExplainResponse
- type FaultRequest
- type HeartbeatRequest
- type JobView
- type JobsResponse
- type KillRequest
- type LabelsPushRequest
- type PollResponse
- type RegisterRequest
- type RegisterResponse
- type Server
- type StateResponse
- type StatusRequest
- type SubmitRequest
- type TTYFrame
- type TTYFrameReader
- type WhoamiResponse
Constants ¶
const ( MinPriority = -10 MaxPriority = 10 )
SubmitRequest deliberately has no lease-TTL field: stage 1 enforces no expiry (watchdogs arrive in a later stage — see the design doc's staging section), and the store's leases.expires_at column is written with an internal default that nothing reads yet. Advertising a lease_ttl_seconds knob on the wire that silently did nothing was worse than not having the knob; the internal column and default stay, ready for the stage that actually enforces them. MinPriority and MaxPriority bound SubmitRequest.Priority. The spec is explicit that priority is "bounded to a small range so it stays a nudge rather than a scheduling language": unbounded values invite callers to encode a policy in the number (1000 for "production", 9999 for "really production"), and since there is no preemption, a big number buys nothing a small one does not — it only makes every later submitter bid higher. Out-of-range submissions are rejected rather than clamped, so a caller who asked for 500 is never quietly given 10 and left believing otherwise.
const ( TTYFrameData = "d" TTYFrameResize = "r" )
The framing for the `in` direction of an interactive session, defined here so the worker end and the client end agree on one wire format rather than each writing their own encoder against the same prose.
The `in` direction carries two kinds of message and is tiny — a keystroke at a time — so it is newline-delimited JSON:
{"t":"d","b":"bHM="} data: base64 keystrokes
{"t":"r","rows":48,"cols":180} resize
The `out` direction has no framing at all: it is high-volume, single-purpose and raw.
The relay itself (tty.go) never decodes any of this. It copies bytes, so a frame type added to this format later reaches a worker through a controller that predates it.
const ControllerInstanceHeader = "Rc-Controller-Instance"
ControllerInstanceHeader carries an identity for the controller PROCESS on every response it writes. A worker that sees it change knows the controller it registered with is gone and a new one is answering, which is the one thing it cannot otherwise detect: the address is the same, the database is the same, and a restart quick enough not to fail a request in flight is invisible from outside.
That blind spot is half of the defect of 2026-08-18. Autonomous recovery evaluates a worker's proof at REGISTRATION, so a worker restart clears a recoverable quarantine and a CONTROLLER restart clears nothing: the worker never re-registered, never presented proof, and the device sat out of the pool until a human cleared it by hand. Seeing this value change is what sends the worker back through registration (see worker.contactTracker).
It is an identity, not a credential. It grants nothing, says nothing about the fleet, and is deliberately stamped on every response including rejections -- a worker whose token was rotated out is still entitled to know it is talking to a new controller.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Assignment ¶
type Assignment struct {
JobID string `json:"job_id"`
DeviceID string `json:"device_id"`
Command []string `json:"command"`
Cwd string `json:"cwd,omitempty"`
Env map[string]string `json:"env,omitempty"`
MaxRuntimeSeconds int `json:"max_runtime_seconds,omitempty"`
IdleTimeoutSeconds int `json:"idle_timeout_seconds,omitempty"`
// Submitter carries the job's submitter so a lease lifecycle hook's
// RC_SUBMITTER can name them.
Submitter string `json:"submitter,omitempty"`
// Kind is model.LeaseKindJob or model.LeaseKindHold. A worker that
// sees "hold" substitutes its own sleeper for Command — see
// internal/worker's execute — rather than trusting what is stored
// here, which for a hold is a fixed, meaningless placeholder.
Kind string `json:"kind,omitempty"`
// Stdio is model.StdioLogs, StdioTTY or StdioPipe. The worker is the
// side that dials out, so this is the only way it ever learns that a job
// wants its stdio spliced onto the relay rather than batched into the
// log store.
Stdio string `json:"stdio,omitempty"`
}
type Config ¶
type Config struct {
Store *store.Store
Logs *logstore.Store
Clock clock.Clock
Tokens map[string]string // token -> role: worker | client | admin
RetainDisconnectedJobs bool
// Notifier delivers operational events (a tripped watchdog, a failed
// verify probe) to whatever sink the operator configured. A nil
// Notifier is the supported and default state — no webhook configured —
// and needs no special handling anywhere: every notify.Notifier method
// is safe on a nil receiver, so emit below simply does nothing.
Notifier *notify.Notifier
}
type DescribeResponse ¶
type DescribeResponse struct {
Device model.Device `json:"device"`
Holder string `json:"holder,omitempty"`
JobID string `json:"job_id,omitempty"`
ElapsedSeconds int `json:"elapsed_seconds"`
HeartbeatAgeSeconds int `json:"heartbeat_age_seconds"`
// QuarantineReason says why this device is out of the pool — see
// DeviceView.QuarantineReason, which this mirrors. Empty when healthy.
QuarantineReason string `json:"quarantine_reason,omitempty"`
Labels []model.Label `json:"labels,omitempty"`
// LabelAgeSeconds is how long ago each label in Labels was last
// confirmed, computed by the controller against its own clock (see
// deviceViews' HeartbeatAgeSeconds, which does the same for the same
// reason: an agent deciding whether to trust a fact must not depend on
// the reader's own clock, which can be skewed — a CLI-side
// time.Now().Sub(UpdatedAt) would let a machine with a fast clock read a
// month-old label as fresh). The absolute timestamp stays on each Label
// too, for a machine consumer that wants it.
//
// Keyed by key+"/"+source. That join is unambiguous even though a label
// KEY may itself contain "/" (labels come from a worker's YAML config or
// a probe script, neither of which constrains the key's charset): the
// necessary and sufficient condition for key+"/"+source to never collide,
// for ANY key content, is that no SOURCE value itself contains "/". (Equal
// length of the source strings, which an earlier version of this comment
// claimed was what mattered, is irrelevant — brute-forced over 599,186
// pairs: a hypothetical third source of "probed", "x", or "nvidia-smi"
// stays collision-free despite different lengths, while one shaped like
// "a/declared" produces 585 collisions despite being unrelated in length
// to the other two. The reason: a collision needs the "/" between some
// key and its source to itself have come FROM a source string, which is
// only possible if some source contains one.) store.ReplaceLabels
// enforces this today by construction — the only two source values it
// accepts, model.SourceDetected ("detected") and model.SourceDeclared
// ("declared"), neither contains "/" — but a future third source value
// MUST preserve "no source may contain a '/'", or this join must switch
// to a delimiter no source can ever contain.
//
// A label absent from this map (Go's comma-ok miss, not a present zero)
// means the controller did not report an age for it — most likely an
// older controller talking to a newer CLI that added this field after
// the controller shipped. Do not render that as "0s ago": that is
// indistinguishable from the freshest possible label and is exactly the
// failure this field exists to prevent, just triggered by version skew
// instead of clock skew. Render it as an explicit unknown instead.
LabelAgeSeconds map[string]int `json:"label_age_seconds,omitempty"`
Sheet string `json:"sheet,omitempty"`
SheetUpdatedAt time.Time `json:"sheet_updated_at,omitempty"`
// SheetAgeSeconds is SheetUpdatedAt's age, computed by the controller
// for the same clock-skew reason as LabelAgeSeconds above. A *int, not a
// plain int, for the same version-skew reason LabelAgeSeconds's map-miss
// carries meaning: an older controller's response simply omits this
// field, which decodes to a nil pointer, distinguishable from a real,
// explicit age of zero. nil also when there is no sheet at all (Sheet is
// "") — not because SheetUpdatedAt happens to be its zero value, which a
// real worker's registration almost never leaves it as (see
// handleDescribe's own host-wide-fallback comment: a real worker sends
// an explicit, non-zero-timestamped empty sheet for a host with no
// host.md), but because an empty sheet has no meaningful age to report
// regardless of what timestamp got attached to recording that emptiness.
SheetAgeSeconds *int `json:"sheet_age_seconds,omitempty"`
// SheetIsHostWide is true when Sheet fell back to the host-wide note
// because this device has none of its own — an agent reading "don't run
// more than two jobs here" needs to know whether that applies to the
// whole box or just this card.
SheetIsHostWide bool `json:"sheet_is_host_wide,omitempty"`
RecentJobs []model.Job `json:"recent_jobs,omitempty"`
// Recoveries are this device's automatic returns to the pool inside the
// flap guard's sliding window, newest first. A device that quarantines,
// clears and quarantines again is describing a real problem, and until
// now nothing outside the store could see that it had happened: a device
// out of the pool looked identical whether it had failed once or had
// spent every automatic return it gets.
Recoveries []DeviceRecovery `json:"recoveries,omitempty"`
// RecoveriesRemaining is how many automatic returns are left inside the
// window. Zero means the next quarantine waits for a person rather than
// clearing itself, which is the difference between a status and a
// decision — so it is reported even when it is the full allowance, and a
// reader never has to know the controller's constant.
//
// A *int for the same version-skew reason as SheetAgeSeconds: an older
// controller omits the field entirely, which decodes to nil and is
// distinguishable from a real zero meaning "none left".
RecoveriesRemaining *int `json:"recoveries_remaining,omitempty"`
// RecoveryWindowSeconds is the width of that sliding window, so a reader
// can say "three times in the last hour" without hardcoding the hour.
RecoveryWindowSeconds int `json:"recovery_window_seconds,omitempty"`
}
DescribeResponse is everything an agent needs to trust (or distrust) a device before writing commands for it: what it is, who holds it now, every label with its provenance and age, the humans' own usage notes and how stale THEY are, and its recent job history.
type DeviceRecovery ¶ added in v0.1.2
type DeviceRecovery struct {
At time.Time `json:"at"`
// AgeSeconds is measured by the controller against its own clock, for
// the same reason every other age on this response is — see
// DescribeResponse.LabelAgeSeconds. The absolute timestamp stays too,
// for a machine consumer that wants it.
AgeSeconds int `json:"age_seconds"`
// Reason is the quarantine this recovery cleared the device FROM.
Reason string `json:"reason,omitempty"`
}
DeviceRecovery is one automatic return to the pool.
type DeviceSpec ¶
type DeviceSpec struct {
Name string `json:"name"`
MaxRuntimeSeconds int `json:"max_runtime_seconds,omitempty"`
}
DeviceSpec is one device a worker declares at registration: its name and, if the operator configured one, the runtime ceiling the controller should enforce against any job scheduled onto it.
type DeviceView ¶
type DeviceView struct {
Device model.Device `json:"device"`
Holder string `json:"holder,omitempty"`
JobID string `json:"job_id,omitempty"`
// Kind is the holding lease's kind (model.LeaseKindJob or
// model.LeaseKindHold), empty when nothing holds the device. Read
// straight off the lease row, which is where task 8 labels a hold —
// see the design note in internal/store/allocate.go's assignQueued.
Kind string `json:"kind,omitempty"`
Reason string `json:"reason,omitempty"`
Command []string `json:"command,omitempty"`
ElapsedSeconds int `json:"elapsed_seconds"`
HeartbeatAgeSeconds int `json:"heartbeat_age_seconds"`
// OldestLabelAgeSeconds is how long ago this device's least-recently-
// confirmed label (across both sources) was last seen, computed by the
// controller against its own clock — see DescribeResponse.LabelAgeSeconds
// for the full reasoning, which applies here verbatim: the dashboard
// (internal/server/dashboard/index.html) used to compute this in the
// BROWSER via Date.parse(label.updated_at) against Date.now(), and its
// own comment documented the resulting exposure honestly — a browser
// clock skewed by more than an hour could show a false staleness
// warning, or hide a real one, on the one age on that page that was not
// already immune to it the way HeartbeatAgeSeconds and ElapsedSeconds
// are. nil when the device has no labels at all (nothing to date); never
// clamped at zero, so a future-stamped label reports a negative value
// rather than being laundered into looking like the freshest possible
// reading — matching formatAge's rule on the CLI side.
OldestLabelAgeSeconds *int `json:"oldest_label_age_seconds,omitempty"`
// QuarantineReason is why this device is out of the pool: the verify
// probe's stderr, a failed acquire hook, `worker_lost`, `registration`,
// or empty when a row was quarantined before reasons were recorded.
// Empty on every healthy device, so its presence alone answers "is
// something wrong here".
//
// It is on the fleet view rather than only on DescribeResponse because
// a page that announces `unhealthy` and offers a "clear" button without
// saying what happened is asking an operator to act on a problem it
// declined to describe — they would have to leave for `rc describe` to
// find out what they were about to return to the pool.
QuarantineReason string `json:"quarantine_reason,omitempty"`
}
type ExplainResponse ¶
type ExplainResponse struct {
Selector string `json:"selector"`
Matching []string `json:"matching"`
Free []string `json:"free"`
QueueDepth int `json:"queue_depth"`
}
ExplainResponse answers "if I submitted this selector right now, what would happen" without actually submitting anything: which devices match, which of those are free this instant, and how backed up the ones that aren't free already are.
type FaultRequest ¶
type FaultRequest struct {
Reason string `json:"reason"`
}
FaultRequest is what a worker sends when it has decided a device must leave the pool: an on_acquire hook that exited non-zero or timed out, or a verify pass that failed after a job (see internal/worker/verify.go).
Reason is free text — the hook's tail output, or the verify pass's "verify failed: ..." — and is NOT persisted on the device row, which has a fixed quarantine_reason vocabulary (see internal/store/reaper.go). Where an operator can actually read it depends on which source produced it, and the two differ:
- a failed hook fails its job too, so the text rides that job's failure report and `rc ps` surfaces it;
- a failed verify pass leaves the job SUCCEEDED (the run was fine; the device is not), so there is no failure report to carry it. It reaches the worker's log, this controller's log (see handleDeviceFault), and the verify_failed webhook event — and nowhere a client API returns.
So: do not describe the job's failure report as the operator-facing "why" in general. It is only that for the hook case.
type HeartbeatRequest ¶
type HeartbeatRequest struct {
RunningJobIDs []string `json:"running_job_ids,omitempty"`
}
HeartbeatRequest is what a worker says on every heartbeat: the IDs of the jobs it is actually supervising right now. The controller renews the leases of exactly those jobs and no others, so a job the worker has no process for — one whose assignment response never arrived, say — stops being renewed and falls to lease expiry rather than being kept alive forever by an unrelated liveness signal.
type JobView ¶
type JobView struct {
Job model.Job `json:"job"`
QueuePosition int `json:"queue_position,omitempty"`
}
JobView is a job plus the queue position a client needs to show progress.
type JobsResponse ¶
type KillRequest ¶
type KillRequest struct {
Submitter string `json:"submitter"`
}
type LabelsPushRequest ¶
type LabelsPushRequest struct {
Host string `json:"host"`
Devices []string `json:"devices"`
Labels map[string]map[string]string `json:"labels"`
DeclaredLabels map[string]map[string]string `json:"declared_labels"`
Sheet *string `json:"sheet,omitempty"`
DeviceSheets map[string]string `json:"device_sheets,omitempty"`
}
LabelsPushRequest is what a worker posts on every probe-interval pass AFTER its initial registration: the same detected/declared facts and usage sheets RegisterRequest carries, scoped to ONLY that. It deliberately has no boot_id or device-upsert path, and handlePushLabels never calls UpsertWorker — see that handler's doc comment for why reusing registration itself for this would be actively dangerous. Sheet has the same nil-vs-empty contract as RegisterRequest.Sheet.
type PollResponse ¶
type PollResponse struct {
Assignments []Assignment `json:"assignments"`
Kills []string `json:"kills,omitempty"`
}
PollResponse is the envelope handleAssignments answers a long-poll with: it carries both newly handed-out assignments and job IDs the controller wants killed, so a kill reaches the worker as fast as an assignment does rather than waiting on a separate channel.
type RegisterRequest ¶
type RegisterRequest struct {
Host string `json:"host"`
BootID string `json:"boot_id,omitempty"`
Devices []DeviceSpec `json:"devices"`
// Recovery is what this worker can prove about processes left behind by
// a job its previous incarnation was running — the cheap sibling of
// BootID, which proves the same thing by having rebooted the machine.
// See model.RecoveryProof and store.AutoRecover.
//
// It is a plain value, not a pointer, and absent means the zero value:
// a worker predating this field, one that could not determine anything,
// and one configured to require a manual clear are indistinguishable
// here on purpose. All three mean "no proof", and no proof means the
// device stays quarantined exactly as it does today. There is no shape
// of this field, and no missing field, that can force a device back
// into the pool.
Recovery model.RecoveryProof `json:"recovery,omitzero"`
// Labels is this registration's freshly detected device facts: the
// empty key "" holds host-wide facts merged into every device (a
// device-scoped value wins any key collision), any other key names one
// device by its bare name.
//
// The field is deliberately NOT json:",omitempty": a nil map (absent
// from the wire entirely) and a non-nil, empty map (present as {})
// decode to different Go values, and that difference is the only signal
// this handler has for telling "the probe pass found nothing at all, so
// say nothing and leave what's stored alone" apart from "the pass ran
// and legitimately detected zero labels, so clear what's stored" — see
// ReplaceLabels's own doc comment for why the latter must replace.
// worker.labelsPayload is the caller-side half of this contract: it
// returns nil, not an empty map, when a whole probe pass produced not a
// single fact, so a fleet-wide probe outage can never wipe every
// device's detected labels and, with them, every selector that depends
// on them.
Labels map[string]map[string]string `json:"labels"`
// DeclaredLabels is the operator-asserted counterpart to Labels (from
// worker.yaml's DeviceConfig.Labels), same shape and the same
// nil-vs-empty contract — though for this source there is no probe to
// fail: the worker always sends its current config verbatim, so an
// operator who removes a declared label and restarts actually sees it
// cleared, the same way an unset max_runtime clears a ceiling.
DeclaredLabels map[string]map[string]string `json:"declared_labels"`
// Sheet is this host's usage-sheet documentation (host.md), and
// DeviceSheets is each device's own (host.d/<device>.md), keyed by bare
// device name.
//
// Sheet is a *string, not a plain string, for the same nil-vs-empty
// reason Labels is a nil-checked map — a fix-round-1 finding: the
// worker's readSheets can fail to read host.md for a reason OTHER than
// it simply not existing (permission denied, ...), and a plain string
// could not tell that apart from "the host genuinely has no sheet".
// nil means "leave whatever is already stored for this host's sheet
// alone"; a non-nil pointer, even to "", is an explicit, trustworthy
// report and is applied via UpsertHostDoc. DeviceSheets needs no such
// pointer: a device whose sheet could not be read is simply omitted
// from the map (its key is absent), which applyDeviceFacts already
// treats as "leave it alone" via its `if body, ok := ...` check.
Sheet *string `json:"sheet,omitempty"`
DeviceSheets map[string]string `json:"device_sheets,omitempty"`
}
type RegisterResponse ¶
type RegisterResponse struct {
WorkerID string `json:"worker_id"`
}
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
type StateResponse ¶
type StateResponse struct {
Devices []DeviceView `json:"devices"`
Jobs []model.Job `json:"jobs"`
Queued []model.Job `json:"queued"`
// QueuedWaitingSeconds is how long each queued job has been waiting,
// keyed by job ID, measured by the controller against its own clock.
//
// A sibling map rather than a field on model.Job: Job is the stored
// shape and every other field on it is a stored value, while this is
// derived at read time. Keeping it beside the list also makes the
// addition purely additive for the existing consumers of `queued`
// (`rc ps` renders it via RenderJobs).
//
// Queued alone is not a useful thing to show — nine seconds is normal,
// forty minutes is a problem — and the wait is computed here for the
// same reason every other age is: a reader's clock must not be able to
// make a stuck queue look fresh.
QueuedWaitingSeconds map[string]int `json:"queued_waiting_seconds,omitempty"`
}
type StatusRequest ¶
type SubmitRequest ¶
type SubmitRequest struct {
DeviceID string `json:"device_id,omitempty"`
// Selector picks a device by its labels instead of by exact ID — give
// exactly one of DeviceID or Selector, never both. See
// store.MatchingDevices for the matching rules.
Selector string `json:"selector,omitempty"`
Command []string `json:"command"`
Cwd string `json:"cwd,omitempty"`
Env map[string]string `json:"env,omitempty"`
Submitter string `json:"submitter"`
IdempotencyKey string `json:"idempotency_key,omitempty"`
Priority int `json:"priority,omitempty"`
MaxRuntimeSeconds int `json:"max_runtime_seconds,omitempty"`
IdleTimeoutSeconds int `json:"idle_timeout_seconds,omitempty"`
NoWait bool `json:"no_wait,omitempty"`
// Kind is model.LeaseKindJob or model.LeaseKindHold; empty means job.
// A hold ("rc hold") is a job whose command the worker chooses for
// itself, never the submitter — see handleSubmit, which rejects a hold
// submission that carries one.
Kind string `json:"kind,omitempty"`
// Reason is why a hold was taken (e.g. "manual profiling"), surfaced by
// rc devices and the dashboard via the lease it is copied onto. Only
// meaningful for a hold.
Reason string `json:"reason,omitempty"`
// Stdio is model.StdioLogs (the default), StdioTTY or StdioPipe: where
// this job's standard streams are wired. The two attached modes put the
// process on the controller's in-memory relay instead of the log store —
// see model.StdioLogs and internal/server/tty.go.
Stdio string `json:"stdio,omitempty"`
}
type TTYFrame ¶
type TTYFrame struct {
T string `json:"t"`
B []byte `json:"b,omitempty"`
Rows uint16 `json:"rows,omitempty"`
Cols uint16 `json:"cols,omitempty"`
}
TTYFrame is one message on the `in` direction. B is base64 on the wire: encoding/json does that for []byte, which is also why the terminal's bytes need no escaping of their own and a frame can never contain a raw newline — what makes newline-delimited framing safe for arbitrary keystrokes.
type TTYFrameReader ¶
type TTYFrameReader struct {
// contains filtered or unexported fields
}
TTYFrameReader reads newline-delimited frames off a stream, reassembling frames that arrive split across chunks — which they will, since the relay copies bytes and respects no message boundary.
func NewTTYFrameReader ¶
func NewTTYFrameReader(r io.Reader) *TTYFrameReader
func (*TTYFrameReader) Next ¶
func (r *TTYFrameReader) Next() (TTYFrame, error)
Next returns the next frame, or io.EOF when the stream ends.
A frame whose type this build does not know is skipped rather than rejected: the format is meant to grow, and an old worker must not kill a terminal because a newer client sent it something extra. Malformed JSON is a different matter — the framing itself is lost — and is an error.
type WhoamiResponse ¶
type WhoamiResponse struct {
Role string `json:"role"`
}
WhoamiResponse tells a caller what its own token can do. It deliberately carries the role and nothing else: not the token, not the token list, not how many tokens exist. It grants nothing — every route still checks the role for itself — so this is a convenience for building a UI that matches the caller's actual powers, never a substitute for those checks.