Documentation
¶
Overview ¶
Package engineagent is the agent half of the control channel: the thing that runs next to an engine on a rented box and tells the control plane that engine exists.
It is deliberately small. Register, sleep, register again. There is no reconnect logic, no backoff state machine and no stream to tend, because a lease has none to have. An agent that crashes and restarts simply registers again, and the control plane reconciles by id rather than by tracking a session. That is a large part of why docs/design/0006 Part 4 chose a lease over a held-open stream.
What is told and what is discovered ¶
The split matters more than the loop does. A box cannot see its own provider identity from inside: hostname is a container id and no provider exports a machine id into the environment (docs/design/0007 finding 4). So the agent's span is partly told to it and partly read by it:
- told, injected at deploy time: engine id, deployment id, provider, host id, node index, endpoint, and where the control plane is
- discovered, read on the box: how many cards are here, and later the link health of issue 213
Failure attribution (issue 214) is only ever as good as what the deploy path stamped, which is why the injection is its own reviewable surface rather than an implementation detail of this package.
CP/DP placement: this runs on the data-plane side and reaches the control plane only through the generated Connect client, which is what CONSTRAINTS CP/DP-1 requires. It never imports internal/provisioners.
Index ¶
- Constants
- Variables
- func AgentPrelude(binaryURL string) (string, error)
- func AgentScript(binaryURL string) (string, error)
- func BinaryURL(arch string) (string, bool)
- func CountCards(ctx context.Context) int32
- func InterconnectImpaired(h *provisionerv1.InterconnectHealth) bool
- func InterconnectProbe(read func(context.Context) *provisionerv1.InterconnectHealth) func(context.Context) bool
- func ReadInterconnect(ctx context.Context) *provisionerv1.InterconnectHealth
- func WrapperScript(binaryURL string, engineCmd []string) (string, error)
- type Agent
- type Identity
- type Option
- func WithCards(n int32) Option
- func WithInterconnect(read func(context.Context) *provisionerv1.InterconnectHealth) Option
- func WithInterval(d time.Duration) Option
- func WithLogger(l *slog.Logger) Option
- func WithProbe(p Probe) Option
- func WithSpan(span []*provisionerv1.EngineNode) Option
- func WithStaging(read func() *provisionerv1.StagingProgress) Option
- type Probe
- type Readiness
- type Registrar
- type StagingSensor
Constants ¶
const ( // EnvEngineID is the correlation key. It is the replica instance id, so // the control plane joins a registration back to the slot that produced // it with a lookup rather than a parse. EnvEngineID = "IPLANE_ENGINE_ID" EnvDeploymentID = "IPLANE_DEPLOYMENT_ID" EnvModel = "IPLANE_ENGINE_MODEL" EnvProvider = "IPLANE_PROVIDER" EnvNodeIndex = "IPLANE_NODE_INDEX" // EnvServiceURL is where to register. The same spelling the other // remote-transport verbs use, so an operator who already exported it // does not learn a second name. EnvServiceURL = "IPLANE_SERVICE_URL" // EnvHealthURL is the engine's own health endpoint, probed over // loopback to separate ASSEMBLING from SERVING. EnvHealthURL = "IPLANE_ENGINE_HEALTH_URL" // EnvEndpoint is the externally reachable engine URL. Normally left // unstamped: on RunPod it derives from a pod id that does not exist // until the create call returns, so the control plane fills it on // registration instead. Honoured when something does know it. EnvEndpoint = "IPLANE_ENGINE_ENDPOINT" // EnvCacheDir is the directory the staging sensor measures. Left unset, // the sensor looks for a Hugging Face cache in the usual places and // reports nothing if it finds none. Set it when the engine was pointed // somewhere else (vLLM's --download-dir, a mounted warm volume). EnvCacheDir = "IPLANE_ENGINE_CACHE_DIR" // EnvHostID is the provider's machine id. Also normally unstamped, for // the same timing reason, and unreadable from inside the container in // any case (docs/design/0007 finding 4). EnvHostID = "IPLANE_HOST_ID" )
Env vars carrying the agent's injected identity.
These names cross a network boundary and a container image: the deploy path stamps them onto the engine container and the agent reads them there. A rename on one side alone produces an engine that registers with no identity, which is the exact failure issue 214's attribution cannot survive.
They live in this package rather than beside either user because both sides are users. `internal/provisioners` writes them and `iplane engine-agent` reads them; the reader owns the vocabulary and the writer imports it, so there is one definition instead of two that agree until they do not.
const DefaultInterval = 5 * time.Second
DefaultInterval is used only until the first successful registration tells the agent what the control plane actually wants. It is deliberately shorter than any plausible lease so a slow first answer does not cost a missed renewal.
const EnvBinaryURL = "IPLANE_AGENT_BINARY_URL"
EnvBinaryURL overrides where the agent binary is fetched from. Set it to run an agent built from a working tree, or to serve the binary from somewhere other than the project's releases.
const RenewDivisor = 3
RenewDivisor is how many renewal attempts fit inside one lease. Three gives two chances before expiry. It mirrors the control plane's own constant; the agent recomputes rather than being told the interval so a control plane that has not been upgraded still yields a sane cadence.
Variables ¶
var DefaultCacheDirs = []string{
"/root/.cache/huggingface",
"/data/.cache/huggingface",
}
DefaultCacheDirs are where a Hugging Face cache lands when nobody said otherwise. Checked in order; the first that exists is measured.
HF_HOME wins when set, and vLLM's --download-dir wins over that, which is why NewStagingSensor takes the directory rather than deciding for itself.
Functions ¶
func AgentPrelude ¶ added in v0.2.4
AgentPrelude returns the shell that fetches the agent and starts it in the background, without saying anything about how the engine is then launched.
Split out because providers disagree about that last part and agree about everything before it. A provider that runs the workload as docker's ENTRYPOINT gets the engine argv plus "$@", because Docker splits ENTRYPOINT and CMD into one argv and the container's own arguments arrive separately. A provider that hands us a standalone startup script has no such split: the whole argv is already in the script and "$@" would be empty. Sharing the fetch and keeping the exec local is the same division the rest of the provider layer uses -- behaviour shared, provider specifics in the adapter.
Everything here is best-effort and swallowed. An engine that can serve tokens must never fail to start because a download failed, a registry was unreachable, or the fetched file would not execute. The symptom of all of those is a member missing from the fleet view, which is the cost we accept; the alternative is a deployment that does not serve because its observability did not install.
func AgentScript ¶
AgentScript returns a /bin/sh script that fetches the agent and runs it in the foreground.
The sidecar counterpart to WrapperScript. There is no engine to exec here and nothing to swallow: the agent IS this container's job, so a failed fetch should exit non-zero and let docker's restart policy retry rather than leaving a container alive doing nothing. That inverts the wrapper's rule, and the reason is that failing here costs no inference, while failing there would have.
func BinaryURL ¶
BinaryURL returns where a container should fetch the agent from, and whether there is anywhere to fetch it from at all.
A dev build has no published artifact. That is reported honestly rather than papered over with a "latest" URL, because a pod pulling an agent from a different version than the control plane it registers with is a class of bug nobody wants to debug from a rented box. An operator who wants an agent from an untagged build points EnvBinaryURL at their own copy.
func CountCards ¶
CountCards reports how many GPUs this node contributes, by asking nvidia-smi.
This is the discovered half of the span (see the package doc). Card count is one of the few things a box genuinely knows about itself, in contrast to its provider identity, which it cannot see at all.
A missing or failing nvidia-smi returns 0 with no error. That is a deliberate choice and the same one issue 213 makes for a sensor-less pool: an agent on a CPU box, or one whose provider does not expose the NVIDIA tooling inside the container, must still register. Reporting zero cards is a legible gap in the fleet view; refusing to register turns a missing reading into a missing member, which is a much worse failure to debug.
func InterconnectImpaired ¶ added in v0.2.4
func InterconnectImpaired(h *provisionerv1.InterconnectHealth) bool
InterconnectImpaired is the threshold, and it is deliberately the simplest one that can be defended: on a board that reports N links, fewer than N up is an impairment.
Link down is unambiguous, and that is what this catches. A link that is up but retrying hard is the interesting middle case and is NOT covered here: reporting it needs per-link error and replay counters, and picking a retry-rate threshold without a real impaired board to calibrate against would be inventing a number. A threshold nobody can defend is worse than a narrower one that is obviously right, because the first thing an operator asks about a degraded member is why.
No reading is never an impairment. Absence of evidence is not evidence of a downed link, and treating it as one would mark every PCIe pool degraded.
func InterconnectProbe ¶ added in v0.2.4
func InterconnectProbe(read func(context.Context) *provisionerv1.InterconnectHealth) func(context.Context) bool
InterconnectProbe adapts the sensor into the impairment half of AnyDegraded, which is where a health check and a link reading combine into one state.
read is injectable so tests, and the mock engine, can supply a reading without an NVLink board present.
func ReadInterconnect ¶ added in v0.2.4
func ReadInterconnect(ctx context.Context) *provisionerv1.InterconnectHealth
ReadInterconnect reports link health for this node by asking nvidia-smi.
Never returns nil, and never reports a fault it did not observe. A missing nvidia-smi, a provider that hides the NVIDIA tooling inside the container, and a PCIe-only box with no links at all are all the same answer: available=false, meaning no reading. That is the same choice CountCards makes for a missing tool, and for the same reason -- a legible gap in the fleet view beats either a false healthy or a hard error, and an agent that refuses to register turns a missing reading into a missing member, which is far worse to debug.
func WrapperScript ¶
WrapperScript returns a /bin/sh script that fetches the agent, starts it in the background, and then hands the container over to the engine.
The shape is the important part. The engine is exec'd, not spawned, so it replaces the shell as the container's main process and the container's lifetime still tracks the engine exactly as it did before. Nothing about the engine's own failure or shutdown behaviour changes because a wrapper was added.
Everything agent-related is best-effort and swallowed. An engine that can serve tokens must never fail to start because a download failed, a registry was unreachable, or the fetched file would not execute. The symptom of all of those is a member missing from the fleet view, which is the cost we accept; the alternative is a deployment that does not serve because its observability did not install, which is indefensible.
engineCmd is the image's real entrypoint (see Deployment.engine_entrypoint on why iplane has to be told). Container args arrive as "$@" and are forwarded verbatim, so the engine sees exactly the arguments it would have seen without the wrapper.
Types ¶
type Agent ¶
type Agent struct {
// contains filtered or unexported fields
}
Agent registers one engine and keeps its lease alive.
func New ¶
New builds an Agent. ident.EngineID is required: without it the registry cannot tell a renewal from a second member.
func (*Agent) Run ¶
Run registers until ctx is cancelled.
A failed registration logs and retries on the next tick. It must not be fatal: the control plane being briefly unreachable is not a reason for a serving engine to stop serving, and the lease already encodes what happens if the outage outlasts it. This is the same reason the binary fetch in the container entrypoint is non-fatal. Nothing about the fleet view is worth costing a request.
type Identity ¶
type Identity struct {
EngineID string
DeploymentID string
Model string
Endpoint string
Provider string
HostID string
NodeIndex int32
}
Identity is everything the agent is told rather than able to read. Every field here arrives from the control plane at deploy time.
HostID is the one worth pausing on. It is the provider's machine id (RunPod's top-level machineId, Vast's machine_id, an EC2 instance id) and it is unreadable from inside the container, so an empty HostID means the deploy path did not stamp it and issue 214 will not be able to attribute a failure to a node. The agent reports the gap rather than inventing a substitute, because a container id that looks like a host id is worse than no host id.
type Option ¶
type Option func(*Agent)
Option configures an Agent.
func WithCards ¶
WithCards sets the GPU count this node contributes. Callers normally pass the result of CountCards.
func WithInterconnect ¶ added in v0.2.4
func WithInterconnect(read func(context.Context) *provisionerv1.InterconnectHealth) Option
WithInterconnect supplies the link-health sensor whose reading separates a group serving at full speed from one serving correctly at a fraction of it.
Injectable rather than always shelling out, for three reasons: tests and the mock engine can report a board that does not exist, an operator on a provider that hides the NVIDIA tooling inside the container can leave it off instead of paying a failing exec every renewal, and a future DCGM-based reader replaces it without touching the agent.
Left unset, the agent makes no interconnect claim at all, which is exactly what an agent built before this existed did.
func WithInterval ¶
WithInterval overrides the starting renewal cadence. The control plane's answer still wins on the first successful registration; this only affects how often the agent retries before it has ever been answered.
func WithProbe ¶
WithProbe supplies the local readiness check that separates ASSEMBLING from SERVING. Without one the agent reports SERVING from the first registration, which is right for an engine with no health endpoint and wrong for everything else.
func WithSpan ¶
func WithSpan(span []*provisionerv1.EngineNode) Option
WithSpan replaces the single-node span the agent would otherwise report about itself.
An agent knows the node it runs on and nothing more, so the default span is that one node. A multi-node engine's real composition comes from the engine, which settles ranks across workers before it serves a token, and one agent then reports the whole group. That path needs a cross-node provider primitive, which docs/design/0006 Part 3 shows no provider we can reach exposes today, so issue 212 is unscheduled and this seam currently has exactly one caller: the mock-engine harness, which fabricates a multi-node span so the fleet view's span column can be exercised without renting a pool.
Naming that honestly matters. This is where the real thing will plug in, and until then it is a demo affordance.
func WithStaging ¶ added in v0.4.0
func WithStaging(read func() *provisionerv1.StagingProgress) Option
WithStaging supplies the sensor that reports weights arriving on this node.
Injectable for the same reasons WithInterconnect is. A test needs to drive a download that is not happening, the mock engine has no cache to watch, and an engine mounting a pre-staged volume should say nothing here rather than reporting that nothing is arriving.
Left unset, the agent makes no staging claim at all.
type Probe ¶
Probe is a local observation of the engine on this box.
It returns a Readiness rather than an error because the agent does not act differently on "not ready" and "the probe itself failed": both mean the group has not formed.
The reason this is a local call matters more than the reason it is pushed. During assembly there is no reachable endpoint for anything outside the machine to ask, so an observer positioned elsewhere cannot distinguish a group that is still forming from one that is slow, broken or gone. An observer inside can watch it come up. Issue 213's link sensor is the same argument applied to a different reading, and plugs in here.
func AnyDegraded ¶ added in v0.2.4
AnyDegraded composes a readiness probe with an impairment probe, which is how a health check and a link sensor combine into one reading.
Not ready wins over degraded: a group that has not formed is not a degraded group, it is a group that is not serving, and reporting the milder state during startup would make assembly look like a fault. Only once the engine answers does an impairment reading get to downgrade it.
impaired is consulted only when serving, so a sensor that is expensive or unavailable during startup is never called then.
func HTTPProbe ¶
HTTPProbe returns a Probe that treats a 2xx from url as Ready.
The agent runs on the box, so this is a loopback call to the engine's own health endpoint and costs nothing worth budgeting. That locality is the point: the control plane's /health poller cannot see ASSEMBLING because during assembly there is no reachable endpoint for it to ask, while the agent is already inside and can watch the engine come up.
It never returns Degraded, and cannot. Health endpoints answer the question they were built for, which is whether the engine is serving, and a degraded group is serving. Detecting that a group is impaired needs a different reading from a different source (issue 213's link sensor); compose the two with AnyDegraded rather than teaching this one to guess.
type Readiness ¶ added in v0.2.4
type Readiness int
Readiness is what a local observation of the engine can conclude. Three values, and the third is the one the chapter exists for.
Deliberately narrower than EngineState. LOST and DRAINING are the control plane's conclusions, one from an expired lease and one from an operator decision, and neither is something an engine can observe about itself. A probe that returned EngineState could express them; this cannot, so the rule is enforced by the type instead of by a check that has to be remembered.
const ( // NotReady means the group has not formed. The engine may be starting, // the workers may still be finding each other, or the probe itself may // have failed; the agent does not distinguish, because all three mean // the same thing to a reader of the fleet view and none of them is // serving. NotReady Readiness = iota // Ready means the engine is answering. Ready // Degraded means the engine is assembled and answering correctly while // running well below what the hardware should give, because something // inside the group is impaired. It is not a failure and it is not // health; a probe built to separate those two has no way to say it. // // A pool with no sensor for this reports Ready, never Degraded. Absence // of a reading is not evidence of a problem, and reporting "degraded" // for every machine we cannot measure would make the state useless. Degraded )
type Registrar ¶
type Registrar interface {
RegisterEngine(context.Context, *connect.Request[provisionerv1.RegisterEngineRequest]) (*connect.Response[provisionerv1.RegisterEngineResponse], error)
}
Registrar is the one control-plane call an agent makes. Narrow on purpose so tests substitute a fake and the real implementation stays the generated Connect client (provisionerv1connect.EngineRegistryServiceClient satisfies this without an adapter).
type StagingSensor ¶ added in v0.4.0
type StagingSensor struct {
// contains filtered or unexported fields
}
StagingSensor measures how fast model weights are landing on this node.
Stateful on purpose: a rate needs two observations, and the agent is the only thing that persists across ticks. Safe for concurrent use, though the agent calls it from one goroutine.
func NewStagingSensor ¶ added in v0.4.0
func NewStagingSensor(dir string) *StagingSensor
NewStagingSensor returns a sensor watching dir, or nil when dir is empty and no default cache directory exists.
A nil sensor is a supported state and reads as "no reading", so a caller need not branch on it beyond passing it along. That matters because the same agent binary runs next to engines that stage weights and engines that were handed a warm volume and stage nothing.
func (*StagingSensor) Dir ¶ added in v0.4.0
func (s *StagingSensor) Dir() string
Dir reports the directory being measured, for the log line that tells an operator which cache the numbers describe. Empty for a nil sensor.
func (*StagingSensor) Read ¶ added in v0.4.0
func (s *StagingSensor) Read() *provisionerv1.StagingProgress
Read reports the bytes on disk and the rate since the previous call.
Never returns nil and never reports a rate it did not measure. An unreadable directory is available=false, the same answer ReadInterconnect gives for a missing nvidia-smi, and for the same reason: a legible gap beats a fabricated zero, and a node whose cache cannot be read must not be mistaken for one whose download has stalled.
The first call establishes the baseline and reports available=true with a zero rate over a zero interval, because one observation genuinely cannot support a rate. Callers distinguish that from a measured stall by looking at interval_seconds rather than at the rate.
A shrinking directory reports a zero rate rather than a negative one. Weights only accumulate during staging, so a decrease means files were evicted or the cache was cleared, and calling that negative throughput would be arithmetic dressed up as a measurement.
func (*StagingSensor) ReadOrNil ¶ added in v0.4.0
func (s *StagingSensor) ReadOrNil() func() *provisionerv1.StagingProgress
ReadOrNil returns Read as a bound function, or nil for a nil sensor, so a caller can hand it to WithStaging without branching.
Read already handles a nil receiver, but handing WithStaging a non-nil function that always reports unavailable would make the agent claim its sensor failed on every renewal. There was nothing to sense, which is a different thing, and the fleet view should say so by staying silent.
Source Files