supervisor

package
v0.1.0-proto2g Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2026 License: MPL-2.0 Imports: 67 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// TargetLocal is reached when the local runtime is up and no network
	// is required (Phase 2).
	TargetLocal = "local.target"

	// TargetNetworkReady is reached when a local agent has published
	// network readiness (Phase 3). This target is what gives that gate a
	// producer - it had none, which is why a nonzero timeout always
	// expired.
	TargetNetworkReady = "network-ready.target"

	// TargetCluster is reached when Serf membership and Raft consensus
	// are up (Phase 4).
	TargetCluster = "cluster.target"

	// TargetDistributed is reached when the scheduler is available
	// (Phase 5).
	TargetDistributed = "distributed.target"
)

Boot targets: the selection mechanism Phase 2 was missing (GOBLIN-DIV-050).

The architecture doc has always described six ordered boot phases with "Phase 2 - Local agents" among them, and nothing started a local agent. The obvious fix - call the kernel's StartAll after discovery - was explicitly rejected by the entry, and the reason is worth restating because it is what shapes this file: discovery registers every agent it finds and enablement defaults to on, so an unfiltered start runs cluster-schedulable templates as node-local processes on every node, AND the scheduler places instances of the same templates. Duplicate execution, measured: the cluster harness installs a sleeper fixture and passes its directory to every node, so StartAll would start a sleeper everywhere.

So the exit is the SELECTION, not the start loop. An agent is boot-local if it is WantedBy a target on the boot path, and cluster-schedulable if it is not. That distinction falls out of the dependency graph the kernel already parses instead of being a classification someone has to invent and keep in sync.

The targets mirror the documented phases and are named for what is TRUE when each is reached, because that is what an agent author has to reason about. Phases 0 and 1 are not attachment points: the agent manager itself comes up in Phase 1, so nothing can be waiting on it.

View Source 
const TagBootstrapExpect = "bootstrap_expect"

TagBootstrapExpect carries a seed node's configured cluster size through gossip. Peers use it to agree on the seed set without an operator designating one node as special.

Variables

View Source 
var ErrInvalidRequest = errors.New("invalid request payload")

ErrInvalidRequest marks a payload the server could not decode into the method's request message. It is a client fault, not a server fault, so it maps to its own code rather than INTERNAL.

View Source 
var ErrMethodNotFound = errors.New("method not found")

ErrMethodNotFound marks a dispatch against a method the server has no handler for. It is a routing miss, not a server fault, so it maps to its own code rather than INTERNAL.

View Source 
var ErrNotAdmitting = errors.New("node never held leadership during the admission window; the leader admits the voter")

ErrNotAdmitting reports that this node never held leadership during the admission window: it is not an error condition - every node sees every Serf join, and only the leader's attempt is supposed to land.

View Source 
var ErrOperatorConfigStale = errors.New("operator key config is stale: the cluster registry does not contain this node's configured keys")

ErrOperatorConfigStale means the cluster's registry does not contain this node's configured keys. It is deliberately NOT fatal.

Once a registry is seeded, --operator-key is inert: it can only succeed as a no-op re-seed of an identical set or be refused, and the enforcement gate reads the replicated registry rather than config. So this condition is a configuration inconsistency, not an operational fault, and killing the node over it would turn untidiness into downtime - or, during a staggered rollout, into an availability incident.

It is also not reliably distinguishable from a lost bootstrap race. hashicorp/raft restores only local state at construction and catches up from the leader afterward, so a brand new node joining a healthy cluster is indistinguishable, from inside this function, from a node that watched the registry fill with someone else's keys. An earlier attempt to tell them apart killed healthy nodes during scale-out. The disagreement is surfaced through the goblin_operator_key_config_drift gauge instead, which an operator can alert on rather than having it scroll past in a log.

The gauge is set once, by the seeder below, and reports the situation as of that check; the seeder is one-shot and nothing re-evaluates it afterwards. Nothing can change the registry in band in piece 1, so the value cannot go stale there. Piece 2's change RPC must re-evaluate it when it commits, or the gauge starts lying.

View Source 
var ErrRevocationsUnavailable = errors.New("no revocation filter on this node")

ErrRevocationsUnavailable is returned by SyncRevocations on a node that carries no revocation filter - nil outside a full supervisor.

A sentinel rather than a formatted string because the sync loop is the caller: a node that can never answer should stop being asked, and that decision needs the refusal to be data.

Functions

func IsPermissionDenied 

func IsPermissionDenied(err error) bool

IsPermissionDenied reports whether err is a server refusal classified PERMISSION_DENIED. It is the client-side half of the contract stated in rpc.proto - callers branch on the code, message text is for humans and is never matched on - and exists so no caller has to reach for strings.Contains to tell a deliberate refusal from a server fault.

func RegisterNodeHandlers 

func RegisterNodeHandlers(server *QUICRPCServer, rpc *NodeRPC)

RegisterNodeHandlers registers node RPC methods.

func RegisterSchedulerHandlers 

func RegisterSchedulerHandlers(server *QUICRPCServer, rpc *SchedulerRPC)

RegisterSchedulerHandlers registers all scheduler RPC methods with the QUIC server

func RegisterSchedulerRPC 

func RegisterSchedulerRPC(sched *scheduler.Scheduler, membership interface{}, consensus *consensus.Consensus) error

RegisterSchedulerRPC registers the scheduler RPC service

Types

type CertManager 

type CertManager struct {
	// contains filtered or unexported fields
}

CertManager serves the control-plane certificate and reloads it from disk when the file changes, so a rotation does not require a restart. One instance backs every plane on the shared listener: the TLS config holds the callbacks, not a snapshot of the key pair.

func NewCertManager 

func NewCertManager(certFile, keyFile string) (*CertManager, error)

func (*CertManager) GetCertificate 

func (cm *CertManager) GetCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error)

func (*CertManager) GetClientCertificate 

func (cm *CertManager) GetClientCertificate(req *tls.CertificateRequestInfo) (*tls.Certificate, error)

func (*CertManager) Load 

func (cm *CertManager) Load() error

func (*CertManager) Watch 

func (cm *CertManager) Watch(ctx context.Context)

Watch polls the certificate file and reloads on a newer mtime. It runs until ctx is cancelled; Supervisor tracks it as a tierRun loop so shutdown joins it.

type Config 

type Config struct {
	NodeID string
	// ListenAddr is the single control-plane bind address: every
	// protocol (gapi-quic, goblin-rpc, serf-quic, raft-quic) shares it,
	// routed by ALPN (GOBLIN-DIV-023). Metrics is the only other port.
	ListenAddr    string
	AdvertiseAddr string
	AdvertisePort int
	RaftDir       string
	// RaftSnapshotThreshold overrides how many outstanding Raft log
	// entries trigger a compaction snapshot (raft.DefaultConfig: 8192).
	// 0 keeps the Raft default; operators tune it down on write-heavy
	// clusters to bound the trailing log's disk footprint and how much a
	// late-joining node must replay before it is caught up.
	RaftSnapshotThreshold uint64
	// RaftSnapshotInterval overrides how often Raft checks whether a
	// compaction snapshot is due (raft.DefaultConfig: 120s). 0 keeps the
	// Raft default.
	RaftSnapshotInterval time.Duration
	// RaftTrailingLogs overrides how many log entries Raft retains after
	// a snapshot for fast follower replay instead of a full snapshot
	// transfer (raft.DefaultConfig: 10240). 0 keeps the Raft default.
	RaftTrailingLogs uint64
	JoinAddr         string
	// BootstrapExpect is the number of seed nodes that must be visible
	// through gossip before the cluster seeds itself. Every seed is
	// configured with the same number and they elect one bootstrapper
	// among themselves, so no node has to be designated by hand. 0 (or
	// 1) keeps the seed model: whichever node has no JoinAddr
	// bootstraps alone.
	BootstrapExpect int
	Tags            map[string]string
	EncryptionKey   string // Base64 encoded 32-byte key
	CertFile        string
	KeyFile         string
	CAFile          string
	MetricsAddr     string
	// ProductionMode restricts embedded-GAPI agent discovery to binaries
	// with verified signatures (review R20).
	ProductionMode bool
	// AgentVerifyKey is the path to the Ed25519 public key that verifies
	// agent-binary signatures; falls back to $GOBLIN_VERIFY_KEY.
	AgentVerifyKey string
	// OperatorKeyFiles are paths to hex-encoded Ed25519 public keys that
	// bootstrap the cluster's operator registry (GOBLIN-DIV-015 piece 1).
	// They are this node's claim about the root of trust; they become
	// authoritative only once committed to Raft. Empty means the cluster
	// refuses every mutating verb.
	OperatorKeyFiles []string
	// Logging mirrors gapi's logging configuration (level, format, file
	// rotation, Loki); handlers are built by the kernel's core/logging.
	Logging gapiconfig.LoggingConfig
	// NetworkGateTimeout bounds the network-readiness phase gate: when
	// nonzero, Run blocks until the kernel's agent.network.running topic
	// fires on the local bus, failing loudly on expiry (GOBLIN-DIV-011,
	// R13). Zero disables the gate - the topic has no producer unless a
	// network agent is deployed.
	NetworkGateTimeout time.Duration
	// Pid1Mode activates the embedded kernel's Phase 0 pre-userspace
	// boot before any cluster code, and the reversed teardown on
	// shutdown (goblin-architecture.md). goblind IS the init process.
	Pid1Mode         bool
	NoEarlyMounts    bool
	WatchdogDevice   string
	WatchdogInterval time.Duration
	ShutdownGrace    time.Duration
}

Config holds configuration for the Supervisor

type LogEvent 

type LogEvent struct {
	Index     uint64
	Timestamp time.Time
	Message   string
}

LogEvent represents a single event in the history

type NodeRPC 

type NodeRPC struct {
	// contains filtered or unexported fields
}

NodeRPC executes the leader's placement decisions on this node: a scheduled instance becomes a real process under the embedded GAPI agent manager (the proto-2 Phase 3 node-dispatch seam, now closed).

func (*NodeRPC) CheckpointAgentInstance 

func (n *NodeRPC) CheckpointAgentInstance(req *goblinv1.NodeCheckpointAgentInstanceRequest, resp *goblinv1.NodeCheckpointAgentInstanceResponse) error

CheckpointAgentInstance dumps a running instance into this node's image store, leaving it stopped.

Stopped is the point. The image is the rollback artifact for the migration, and a source that kept running past the state its image captured would have diverged from it before the destination even started restoring.

func (*NodeRPC) Images 

func (n *NodeRPC) Images() *migration.Store

Images is this node's checkpoint image store.

func (*NodeRPC) MigrationReady 

func (n *NodeRPC) MigrationReady(req *goblinv1.NodeMigrationReadyRequest, resp *goblinv1.NodeMigrationReadyResponse) error

MigrationReady is the pre-flight the coordinator runs against a prospective destination BEFORE checkpointing the source (GOBLIN-DIV-048).

Not ready is reported as a populated response, not an error: an unready node and an unreachable one are different facts and the coordinator refuses differently for each. The raft indices ride along because "not caught up" and "caught up but empty" want opposite fixes, and a bare boolean cannot tell them apart - which is exactly what cost this entry two rounds of inference.

func (*NodeRPC) PullCheckpoint 

func (n *NodeRPC) PullCheckpoint(req *goblinv1.NodePullCheckpointRequest, resp *goblinv1.NodePullCheckpointResponse) error

PullCheckpoint fetches an instance's image from a peer into this node's store.

This runs on the DESTINATION. The coordinator on the leader tells it where to pull from rather than relaying the bytes: the leader is frequently neither end of the transfer, and routing a multi-gigabyte image through the node running consensus is exactly what the separate goblin-ckpt ALPN exists to avoid.

func (*NodeRPC) RestoreAgentInstance 

func (n *NodeRPC) RestoreAgentInstance(req *goblinv1.NodeRestoreAgentInstanceRequest, resp *goblinv1.NodeRestoreAgentInstanceResponse) error

RestoreAgentInstance restores an instance from an image in this node's store and adopts the resulting process.

The identity captured afterwards is what makes the migration visible to the rest of the cluster: the restored process has a new pid, a new pid namespace inode and a new start epoch, and publishing them through the ordinary heartbeat is the locator move. The instance UUID does not change - that is the entire migration semantic.

func (*NodeRPC) SignalAgentInstance 

func (n *NodeRPC) SignalAgentInstance(req *goblinv1.NodeSignalAgentInstanceRequest, resp *goblinv1.NodeSignalAgentInstanceResponse) error

SignalAgentInstance delivers a signal through the start-epoch + pidfd guard. The request was already authorized at the leader's FSM; this node only guards delivery: a stale epoch means the process the caller meant is gone, so the delivery is refused with no retry.

func (*NodeRPC) StartAgentInstance 

func (n *NodeRPC) StartAgentInstance(req *goblinv1.NodeStartAgentInstanceRequest, resp *goblinv1.NodeStartAgentInstanceResponse) error

StartAgentInstance instantiates the spec's agent type - which must be installed and discovery-verified on this node - and starts it under the instance id. Specs reference installed types; they never carry commands, so the discovery security model (R20) holds for scheduled work.

func (*NodeRPC) StopAgentInstance 

func (n *NodeRPC) StopAgentInstance(req *goblinv1.NodeStopAgentInstanceRequest, resp *goblinv1.NodeStopAgentInstanceResponse) error

StopAgentInstance stops and deregisters an instance. Unknown instances succeed: a stop for something already gone is the desired state.

type QUICRPCClient 

type QUICRPCClient struct {
	// contains filtered or unexported fields
}

QUICRPCClient is a client for making RPC calls over QUIC

func NewQUICRPCClient 

func NewQUICRPCClient(addr string, tlsConfig *tls.Config) (*QUICRPCClient, error)

NewQUICRPCClient creates a new QUIC RPC client

func (*QUICRPCClient) Call 

func (c *QUICRPCClient) Call(method string, req, resp proto.Message) error

Call sends a protobuf request and decodes a protobuf response. The payload inside the envelope is a generated message, so buf breaking covers every field that crosses the wire (GOBLIN-DIV-036).

func (*QUICRPCClient) Close 

func (c *QUICRPCClient) Close() error

Close closes the QUIC connection

type QUICRPCServer 

type QUICRPCServer struct {
	// contains filtered or unexported fields
}

QUICRPCServer serves RPC over QUIC

func NewQUICRPCServer 

func NewQUICRPCServer() *QUICRPCServer

NewQUICRPCServer creates a new QUIC RPC server

func (*QUICRPCServer) HandleConnection 

func (s *QUICRPCServer) HandleConnection(conn *quic.Conn)

HandleConnection handles a single QUIC connection

func (*QUICRPCServer) RegisterHandler 

func (s *QUICRPCServer) RegisterHandler(method string, handler RPCHandler)

RegisterHandler registers an RPC method handler

type RPCCallError 

type RPCCallError struct {
	Code    goblinv1.RPCErrorCode
	Message string
}

RPCCallError is the client-side error carrying a server RPCError. Callers branch on Code via errors.As; Message is for humans.

func (*RPCCallError) Error 

func (e *RPCCallError) Error() string

type RPCHandler 

type RPCHandler func(payload []byte) ([]byte, error)

RPCHandler processes RPC requests and returns responses

type SchedulerRPC 

type SchedulerRPC struct {
	// contains filtered or unexported fields
}

SchedulerRPC exposes scheduler operations via RPC

func (*SchedulerRPC) AddEvent 

func (s *SchedulerRPC) AddEvent(msg string)

AddEvent appends a regular log message to the history

func (*SchedulerRPC) DeleteGlobalAgent 

func (s *SchedulerRPC) DeleteGlobalAgent(req *goblinv1.DeleteGlobalAgentRequest, resp *goblinv1.DeleteGlobalAgentResponse) error

DeleteGlobalAgent removes a global agent spec.

func (*SchedulerRPC) DrainNode 

func (s *SchedulerRPC) DrainNode(req *goblinv1.DrainNodeRequest, resp *goblinv1.DrainNodeResponse) error

DrainNode handles node draining via RPC.

func (*SchedulerRPC) GetEvents 

func (s *SchedulerRPC) GetEvents(req *goblinv1.GetEventsRequest, resp *goblinv1.GetEventsResponse) error

GetEvents returns events occurring after the given cursor

func (*SchedulerRPC) GetGlobalAgent 

func (s *SchedulerRPC) GetGlobalAgent(req *goblinv1.GetGlobalAgentRequest, resp *goblinv1.GetGlobalAgentResponse) error

GetGlobalAgent returns a specific global agent by ID

func (*SchedulerRPC) ListAgentInstances 

func (s *SchedulerRPC) ListAgentInstances(req *goblinv1.ListAgentInstancesRequest, resp *goblinv1.ListAgentInstancesResponse) error

ListAgentInstances returns the scheduler's instance records. SpecId accepts either the canonical spec UUID or the operator-facing name.

func (*SchedulerRPC) ListGlobalAgents 

func (s *SchedulerRPC) ListGlobalAgents(req *goblinv1.ListGlobalAgentsRequest, resp *goblinv1.ListGlobalAgentsResponse) error

ListGlobalAgents returns all global agents

func (*SchedulerRPC) ListJobs 

func (s *SchedulerRPC) ListJobs(req *goblinv1.ListJobsRequest, resp *goblinv1.ListJobsResponse) error

ListJobs returns all jobs in the cluster

func (*SchedulerRPC) ListLocalAgents 

func (s *SchedulerRPC) ListLocalAgents(req *goblinv1.ListLocalAgentsRequest, resp *goblinv1.ListLocalAgentsResponse) error

ListLocalAgents returns agents managed by the local GAPI agent manager

func (*SchedulerRPC) Members 

func (s *SchedulerRPC) Members(req *goblinv1.MembersRequest, resp *goblinv1.MembersResponse) error

Members returns the list of cluster members

func (*SchedulerRPC) MigrateInstance 

func (s *SchedulerRPC) MigrateInstance(req *goblinv1.MigrateInstanceRequest, resp *goblinv1.MigrateInstanceResponse) error

MigrateInstance live-migrates a running instance to another node.

func (*SchedulerRPC) MigrateJob 

func (s *SchedulerRPC) MigrateJob(req *goblinv1.MigrateJobRequest, resp *goblinv1.MigrateJobResponse) error

MigrateJob handles job migration via RPC.

func (*SchedulerRPC) PublishEvent 

func (s *SchedulerRPC) PublishEvent(req *goblinv1.PublishEventRequest, resp *goblinv1.PublishEventResponse) error

PublishEvent publishes an event to the cluster via the EventBus. membership is interface{} on SchedulerRPC to avoid an import cycle, so UserEvent support is checked with a local interface rather than a concrete type.

func (*SchedulerRPC) RegisterGlobalAgent 

func (s *SchedulerRPC) RegisterGlobalAgent(req *goblinv1.RegisterGlobalAgentRequest, resp *goblinv1.RegisterGlobalAgentResponse) error

RegisterGlobalAgent registers a new global agent spec. spec_uuid is server-owned: RegisterAgent mints it from spec.name when unset. A caller-supplied spec_uuid is rejected rather than silently overwritten (design doc, "Server-owned fields must be rejected, not overwritten") so a client bug cannot masquerade as the minted identity.

func (*SchedulerRPC) ScaleAgent 

func (s *SchedulerRPC) ScaleAgent(req *goblinv1.ScaleAgentRequest, resp *goblinv1.ScaleAgentResponse) error

ScaleAgent updates the replica count for an agent.

func (*SchedulerRPC) SignalAgentInstance 

func (s *SchedulerRPC) SignalAgentInstance(req *goblinv1.SignalAgentInstanceRequest, resp *goblinv1.SignalAgentInstanceResponse) error

SignalAgentInstance is the leader-side signal path (GOBLIN-DIV-017, DDR-10): issue a capability token for exactly the required right, verify it through the kernel's single verification codepath (with the revocation filter - defense in depth), commit the request through Raft where the FSM authorizes it against the rights bitmap and answers with the placement node, then dispatch delivery to that node's epoch-guarded pidfd path.

func (*SchedulerRPC) SubmitJob 

func (s *SchedulerRPC) SubmitJob(req *goblinv1.SubmitJobRequest, resp *goblinv1.SubmitJobResponse) error

SubmitJob handles job submission via RPC.

func (*SchedulerRPC) SyncRevocations 

func (s *SchedulerRPC) SyncRevocations(req *goblinv1.SyncRevocationsRequest, resp *goblinv1.SyncRevocationsResponse) error

SyncRevocations is the anti-entropy exchange that repairs revocations the best-effort delta broadcast dropped (GOBLIN-DIV-057).

It is symmetric on purpose: the caller sends its live generations and receives the responder's, so one round trip repairs both nodes. No leader and no ordering are needed - merging is idempotent and the filter is a set, so any node may sync with any other at any time.

type Supervisor 

type Supervisor struct {
	// contains filtered or unexported fields
}

Supervisor manages the Goblin daemon components

func New 

func New(cfg Config) *Supervisor

New creates a new Supervisor

func (*Supervisor) Run 

func (s *Supervisor) Run(ctx context.Context) (err error)

Run boots the node through its phases and blocks until the context is cancelled, then tears down in a bounded, ordered sequence.

The phase order here IS the boot contract. It differs from what the architecture doc described before this change: the local runtime and the network-readiness gate now precede cluster join, rather than following it. Phase 2 (local agent start) is absent - see GOBLIN-DIV-050 - so the gate has no producer and a nonzero NetworkGateTimeout always expires.

There is exactly ONE exit path. boot returns on the first phase error or on cancellation; teardown then runs unconditionally, in one order, for both cases. Resource shutdowns are deliberately NOT defers: a defer registered when a resource is created runs in creation order, which puts consensus and membership teardown BEFORE the loops that touch them have been joined. That inversion is the defect GOBLIN-DIV-038 names, and it cannot be fixed by reordering defers because the join point has to outlive every one of them.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.