backend

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 16, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package backend abstracts where a job's container actually runs. The controller speaks only to this interface, so the same submit/run/cancel logic drives local Docker today (docker.go) and Kubernetes later (P6) without change.

Index

Constants

View Source 
const (
	// PullIfNotPresent pulls only when the image is absent locally. Default.
	// Local-only images (e.g. a dev-built runner tag) are never pulled.
	PullIfNotPresent = "ifnotpresent"
	// PullAlways always attempts a pull before launch.
	PullAlways = "always"
	// PullNever never pulls; the image must already be present.
	PullNever = "never"
)

Pull policies for LaunchSpec.PullPolicy.

Variables

This section is empty.

Functions

This section is empty.

Types

type CapacityConfig 

type CapacityConfig struct {
	MaxJobs          int
	DefaultJobCPU    string
	DefaultJobMemory string
}

CapacityConfig describes Weibo's scheduling policy for converting backend resources into user-facing job slots.

type CapacitySnapshot 

type CapacitySnapshot struct {
	Backend string    `json:"backend"`
	Health  string    `json:"health"`
	Source  string    `json:"source,omitempty"`
	Reason  string    `json:"reason,omitempty"`
	At      time.Time `json:"observedAt"`

	TotalSlots     *int `json:"totalSlots,omitempty"`
	UsedSlots      int  `json:"usedSlots"`
	AvailableSlots *int `json:"availableSlots,omitempty"`
	MaxJobs        int  `json:"maxJobs,omitempty"`

	CPUTotalMilli     int64 `json:"cpuTotalMilli,omitempty"`
	CPUReservedMilli  int64 `json:"cpuReservedMilli,omitempty"`
	CPUAvailableMilli int64 `json:"cpuAvailableMilli,omitempty"`

	MemoryTotalBytes     int64 `json:"memoryTotalBytes,omitempty"`
	MemoryReservedBytes  int64 `json:"memoryReservedBytes,omitempty"`
	MemoryAvailableBytes int64 `json:"memoryAvailableBytes,omitempty"`

	DefaultJobCPUMilli    int64            `json:"defaultJobCPUMilli,omitempty"`
	DefaultJobMemoryBytes int64            `json:"defaultJobMemoryBytes,omitempty"`
	RunningContainers     int              `json:"runningContainers"`
	StartingContainers    int              `json:"startingContainers"`
	ExitedContainers      int              `json:"exitedContainers"`
	UnhealthyContainers   int              `json:"unhealthyContainers"`
	Unsupported           bool             `json:"unsupported,omitempty"`
	Host                  *HostStats       `json:"host,omitempty"`
	Containers            []ContainerStats `json:"containers,omitempty"`
}

CapacitySnapshot is a backend-normalized view of how many Weibo job containers can run now. Nil slot values mean the backend cannot determine them accurately.

type ContainerBackend 

type ContainerBackend interface {
	// Launch starts a container and returns its backend-specific ID. The
	// data volume is reused across launches of the same JobID so state
	// and checkpoints survive a restart.
	Launch(ctx context.Context, spec LaunchSpec) (containerID string, err error)
	// Stop requests a graceful stop (SIGTERM), waiting up to timeout
	// before killing. Safe to call on an already-stopped container.
	Stop(ctx context.Context, containerID string, timeout time.Duration) error
	// Status reports the container's current phase and control address.
	Status(ctx context.Context, containerID string) (Status, error)
	// Logs returns up to tail lines of the container's stdout+stderr.
	Logs(ctx context.Context, containerID string, tail int) (string, error)
	// Remove deletes the container (not its data volume).
	Remove(ctx context.Context, containerID string) error
	// Capacity reports backend health and job-slot capacity.
	Capacity(ctx context.Context, cfg CapacityConfig) (CapacitySnapshot, error)
}

ContainerBackend launches and manages one container per job.

type ContainerStats added in v0.9.0 

type ContainerStats struct {
	ID               string  `json:"id"`
	Name             string  `json:"name"`
	JobID            string  `json:"jobId,omitempty"`
	Managed          bool    `json:"managed"`
	Image            string  `json:"image,omitempty"`
	ImageID          string  `json:"imageId,omitempty"`
	State            string  `json:"state"`
	CPUPercent       float64 `json:"cpuPercent"`
	MemoryUsedBytes  uint64  `json:"memoryUsedBytes"`
	MemoryLimitBytes uint64  `json:"memoryLimitBytes"`
	MemoryPercent    float64 `json:"memoryPercent"`
	NetworkRxBytes   uint64  `json:"networkRxBytes"`
	NetworkTxBytes   uint64  `json:"networkTxBytes"`
	BlockReadBytes   uint64  `json:"blockReadBytes"`
	BlockWriteBytes  uint64  `json:"blockWriteBytes"`
	PIDs             uint64  `json:"pids"`
	StartedAt        int64   `json:"startedAt,omitempty"`
}

ContainerStats is a live compute snapshot for one Weibo-managed container.

type Docker 

type Docker struct {
	// contains filtered or unexported fields
}

Docker runs each job as a local Docker container using the runner image. The workflow document is copied into the container before start (so it works regardless of where the daemon runs), and /data is backed by a named volume per job so state and checkpoints survive restarts.

func NewDocker 

func NewDocker(image string) (*Docker, error)

NewDocker connects to the Docker daemon from the environment. image is the runner image tag to launch (e.g. "weibo-runner:dev").

func (*Docker) Capacity 

func (d *Docker) Capacity(ctx context.Context, cfg CapacityConfig) (CapacitySnapshot, error)

func (*Docker) HasImage 

func (d *Docker) HasImage(ctx context.Context, ref string) (bool, error)

HasImage reports whether the given image reference is present locally, so the CLI can fail fast with a build hint instead of a launch error.

func (*Docker) Launch 

func (d *Docker) Launch(ctx context.Context, spec LaunchSpec) (string, error)

func (*Docker) Logs 

func (d *Docker) Logs(ctx context.Context, id string, tail int) (string, error)

func (*Docker) Ping 

func (d *Docker) Ping(ctx context.Context) error

Ping verifies the daemon is reachable — used at controller startup and to gate integration tests.

func (*Docker) Remove 

func (d *Docker) Remove(ctx context.Context, id string) error

func (*Docker) Status 

func (d *Docker) Status(ctx context.Context, id string) (Status, error)

func (*Docker) Stop 

func (d *Docker) Stop(ctx context.Context, id string, timeout time.Duration) error

type Fake 

type Fake struct {

	// LaunchErr, if set, makes the next Launch fail (then clears).
	LaunchErr error
	// contains filtered or unexported fields
}

Fake is an in-memory ContainerBackend for controller unit tests. It records launches and lets a test drive each container's phase without Docker. Safe for concurrent use.

func NewFake 

func NewFake() *Fake

NewFake returns a ready fake backend.

func (*Fake) Capacity 

func (f *Fake) Capacity(ctx context.Context, cfg CapacityConfig) (CapacitySnapshot, error)

func (*Fake) LastEnv 

func (f *Fake) LastEnv(id string) map[string]string

LastEnv returns the env a container was launched with (for assertions).

func (*Fake) LastImage 

func (f *Fake) LastImage(id string) string

LastImage returns the image a container was launched with.

func (*Fake) LastResources 

func (f *Fake) LastResources(id string) *ResourceLimits

LastResources returns the resource limits a container was launched with (nil when the job set none).

func (*Fake) LastWorkflowDoc 

func (f *Fake) LastWorkflowDoc(id string) []byte

LastWorkflowDoc returns the workflow document a container was launched with (empty for SDK jobs).

func (*Fake) Launch 

func (f *Fake) Launch(ctx context.Context, spec LaunchSpec) (string, error)

func (*Fake) Launched 

func (f *Fake) Launched() int

Launched reports how many containers were ever launched.

func (*Fake) Logs 

func (f *Fake) Logs(ctx context.Context, id string, tail int) (string, error)

func (*Fake) Remove 

func (f *Fake) Remove(ctx context.Context, id string) error

func (*Fake) SetLogs 

func (f *Fake) SetLogs(id, logs string)

SetLogs sets the log text a container returns.

func (*Fake) SetPhase 

func (f *Fake) SetPhase(id string, phase Phase, exitCode int)

SetPhase forces a container's phase, simulating an exit or crash.

func (*Fake) Status 

func (f *Fake) Status(ctx context.Context, id string) (Status, error)

func (*Fake) Stop 

func (f *Fake) Stop(ctx context.Context, id string, timeout time.Duration) error

type HostStats added in v0.9.0 

type HostStats struct {
	Hostname         string  `json:"hostname,omitempty"`
	OperatingSystem  string  `json:"operatingSystem,omitempty"`
	Architecture     string  `json:"architecture,omitempty"`
	KernelVersion    string  `json:"kernelVersion,omitempty"`
	DockerVersion    string  `json:"dockerVersion,omitempty"`
	CPUCores         int     `json:"cpuCores"`
	CPUPercent       float64 `json:"cpuPercent"`
	MemoryUsedBytes  int64   `json:"memoryUsedBytes"`
	MemoryTotalBytes int64   `json:"memoryTotalBytes"`
	MemoryPercent    float64 `json:"memoryPercent"`
	Load1            float64 `json:"load1"`
	Load5            float64 `json:"load5"`
	Load15           float64 `json:"load15"`
}

HostStats is live utilization for the machine running the Docker daemon.

type Kubernetes 

type Kubernetes struct{}

Kubernetes is a placeholder when the binary is built without the kubernetes build tag.

func NewKubernetes 

func NewKubernetes(opts KubernetesOptions) (*Kubernetes, error)

NewKubernetes reports how to build a Kubernetes-enabled controller.

func (*Kubernetes) Capacity 

func (k *Kubernetes) Capacity(ctx context.Context, cfg CapacityConfig) (CapacitySnapshot, error)

func (*Kubernetes) Launch 

func (k *Kubernetes) Launch(ctx context.Context, spec LaunchSpec) (string, error)

func (*Kubernetes) Logs 

func (k *Kubernetes) Logs(ctx context.Context, containerID string, tail int) (string, error)

func (*Kubernetes) Ping 

func (k *Kubernetes) Ping(ctx context.Context) error

func (*Kubernetes) Remove 

func (k *Kubernetes) Remove(ctx context.Context, containerID string) error

func (*Kubernetes) Status 

func (k *Kubernetes) Status(ctx context.Context, containerID string) (Status, error)

func (*Kubernetes) Stop 

func (k *Kubernetes) Stop(ctx context.Context, containerID string, timeout time.Duration) error

type KubernetesOptions 

type KubernetesOptions struct {
	Kubeconfig       string
	Namespace        string
	Image            string
	PVCSize          string
	StorageClass     string
	ImagePullSecrets []string
}

KubernetesOptions configures the Kubernetes backend. The default binary is built without Kubernetes support to keep Docker-only installs lightweight.

type LaunchSpec 

type LaunchSpec struct {
	JobID string // used to name the container and its data volume
	Name  string // human-readable workflow name
	Image string // runner image, e.g. weibo-runner:dev

	// WorkflowDoc is the raw workflow file content. The backend makes it
	// available to the container and points WORKFLOW at it.
	WorkflowDoc []byte

	// Env is the resolved runtime environment (including any secrets).
	// The backend passes it to the container; it is never persisted.
	Env map[string]string

	// ControlPort is the container port the jobagent listens on. The
	// backend publishes it and reports the reachable host port in Status.
	ControlPort int

	// RestoreSavepoint, if set, names a savepoint the runner seeds from
	// before starting (RESTORE_SAVEPOINT). Empty means a fresh start.
	RestoreSavepoint string

	// PullPolicy controls whether the backend pulls Image before launch:
	// PullAlways, PullNever, or PullIfNotPresent (the default when empty).
	PullPolicy string

	// Resources caps the container's CPU/memory. Nil means unlimited
	// (today's behavior); each backend maps it to its native limits.
	Resources *ResourceLimits
}

LaunchSpec is everything the backend needs to start one job container.

type Phase 

type Phase string

Phase is the coarse container lifecycle as the backend sees it — distinct from the richer job lifecycle the controller tracks. 

const (
	// PhaseRunning: the container process is up.
	PhaseRunning Phase = "running"
	// PhaseExited: the container process has stopped (see ExitCode).
	PhaseExited Phase = "exited"
	// PhaseGone: no such container (never launched, or removed).
	PhaseGone Phase = "gone"
)

type ResourceLimits 

type ResourceLimits struct {
	CPU    string
	Memory string
}

ResourceLimits are CPU/memory caps expressed as Kubernetes quantity strings (e.g. CPU "500m" or "2", Memory "512Mi" or "1Gi"). An empty field means "no limit for this dimension". The controller validates the strings before launch, so backends can assume they parse.

type Status 

type Status struct {
	Phase    Phase
	ExitCode int    // meaningful when Phase == PhaseExited
	HostPort int    // reachable host port mapped to ControlPort, 0 if none
	Address  string // host:port for the control surface, empty if unreachable
}

Status is a point-in-time container status.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.