engine

Documentation

Overview

Package engine plans and launches Exedra model executors as local subprocesses. Supervision policy belongs to the callers.

Example

package main

import (
	"context"
	"log"
	"os"

	"altlinux.space/exedra/exedra-engine/pkg/engine"
)

func main() {
	hw, err := engine.Probe()
	if err != nil {
		log.Fatal(err)
	}

	spec := engine.ModelSpec{
		ID:   "qwen",
		Path: "/var/lib/exedra/models/qwen/current/model.gguf",
		Kind: engine.KindLlama,
	}

	// admin overrides are just another layer, the engine merges them
	cfg, err := engine.LoadConfig(engine.DefaultConfigPath)
	if err != nil {
		log.Fatal(err)
	}
	con := engine.Constraints{APIKey: "secret", Overrides: cfg}

	plan, err := engine.Plan(spec, hw, con)
	if err != nil {
		log.Fatal(err)
	}

	inst, err := engine.Launch(context.Background(), plan, os.Stderr)
	if err != nil {
		log.Fatal(err)
	}
	log.Print(inst.Endpoint().URL)

	// the stop deadline must outlive the context that cancelled the caller
	stopCtx, cancel := context.WithTimeout(context.Background(), engine.StopTimeout)
	defer cancel()
	if err := inst.Stop(stopCtx); err != nil {
		log.Print(err)
	}
}

Index

• Constants
• Variables
• type CPU
• type Config
  • func LoadConfig(path string) (*Config, error)
• type Constraints
• type Endpoint
• type ExecutorInfo
• type GPU
• type Hardware
  • func Probe() (*Hardware, error)
  • func (h *Hardware) Executor(kind Kind) (ExecutorInfo, bool)
  • func (h *Hardware) HasGPU() bool
• type Instance
  • func Launch(ctx context.Context, plan *LaunchPlan, logs io.Writer) (*Instance, error)
  • func (i *Instance) Endpoint() Endpoint
  • func (i *Instance) ExitReason() error
  • func (i *Instance) Pid() int
  • func (i *Instance) Plan() *LaunchPlan
  • func (i *Instance) Stop(ctx context.Context) error
  • func (i *Instance) Wait() <-chan struct{}
• type Kind
  • func ParseKind(s string) (Kind, error)
  • func SupportedKinds() []Kind
• type LaunchPlan
  • func Plan(spec ModelSpec, hw *Hardware, c Constraints) (*LaunchPlan, error)
  • func (p *LaunchPlan) URL() string
• type ModelInfo
  • func Inspect(path string) (*ModelInfo, error)
• type ModelSpec
• type Override

Examples

• Package

Constants

const (
	KindLlama      = executor.Llama
	KindTranscribe = executor.Transcribe
)

Supported executor kinds (the exedra.engine field of a model).

const DefaultHost = "127.0.0.1"

DefaultHost is the bind address used when Constraints.Host is empty.

const DefaultPortBase = 18080

DefaultPortBase is the first port tried when Constraints.Port is 0.

const StopTimeout = 10 * time.Second

StopTimeout is the SIGTERM grace period suggested for Stop contexts.

Variables

var (
	ErrUnsupportedKind  = executor.ErrUnsupported
	ErrExecutorNotFound = errors.New("executor is not installed")
	ErrNoHardware       = errors.New("hardware snapshot is nil")
	ErrNotReady         = errors.New("executor did not become ready")
	ErrNotGGUF          = errors.New("not a GGUF file")
	ErrNoEndpoint       = errors.New("args and env set no listen address")
)

Sentinel errors for callers to match with errors.Is.

var DefaultConfigPath = "/etc/exedra/engine.d"

DefaultConfigPath is the system-wide override directory; build systems override it via -ldflags to follow their sysconfdir.

Types

type CPU

type CPU struct {
	Threads int   `json:"threads"`
	RAM     int64 `json:"ram"` // bytes, 0 if unknown
}

CPU describes the host processor and memory.

type Config

type Config struct {
	Env     map[string]string   `yaml:"env" json:"env,omitempty"`
	Engines map[Kind]Override   `yaml:"engines" json:"engines,omitempty"`
	Models  map[string]Override `yaml:"models" json:"models,omitempty"`
}

Config holds launch overrides, lowest priority first: Env for every executor, Engines by executor kind, Models by model id. Args are executor-specific, so there is no global args layer.

func LoadConfig

func LoadConfig(path string) (*Config, error)

LoadConfig reads the YAML overrides from a directory (*.yaml in file name order, a later file overrides an earlier one) or from one file; "" means DefaultConfigPath and a missing path is an empty config.

type Constraints

type Constraints struct {
	Host      string   // bind address; "" = DefaultHost
	Ctx       int      // context size; 0 = executor default
	Threads   int      // CPU threads; 0 = executor default
	Port      int      // exact port; 0 = scan from PortBase
	PortBase  int      // first port of the scan; 0 = DefaultPortBase
	APIKey    string   // "" leaves the executor unauthenticated
	NoGPU     bool     // force CPU even when a GPU is present
	UI        bool     // keep the executor web UI enabled
	Overrides *Config  // admin overrides, applied by model id; nil = none
	ExtraArgs []string // caller flags, the top layer
	ExtraEnv  []string // caller environment, KEY=VALUE, the top layer
}

Constraints tune planning; zero values are derived or left to the executor.

type Endpoint

type Endpoint struct {
	URL    string `json:"url"`
	APIKey string `json:"api_key"`
}

Endpoint is the address and API key of a ready instance.

type ExecutorInfo

type ExecutorInfo struct {
	Kind     Kind     `json:"kind"`
	Path     string   `json:"path"`
	Backends []string `json:"backends,omitempty"`
}

ExecutorInfo is one installed engine binary and its backend plugins.

type GPU

type GPU struct {
	Card   string `json:"card"`
	Driver string `json:"driver"`
	VRAM   int64  `json:"vram"`
}

GPU is one DRM render device.

type Hardware

type Hardware struct {
	CPU       CPU            `json:"cpu"`
	GPUs      []GPU          `json:"gpus"`
	Executors []ExecutorInfo `json:"executors"`
}

Hardware is a host snapshot used for planning.

func Probe

func Probe() (*Hardware, error)

Probe inspects the host: CPU/RAM, DRM GPUs and installed executors.

func (*Hardware) Executor

func (h *Hardware) Executor(kind Kind) (ExecutorInfo, bool)

Executor finds the installed binary for a kind.

func (*Hardware) HasGPU

func (h *Hardware) HasGPU() bool

HasGPU reports whether any render device was found.

type Instance

type Instance struct {
	// contains filtered or unexported fields
}

Instance is one running executor process.

func Launch

func Launch(ctx context.Context, plan *LaunchPlan, logs io.Writer) (*Instance, error)

Launch resolves the endpoint from the plan's Args and Env, starts the process and waits until ready. It retains plan: do not modify it afterwards.

func (*Instance) Endpoint

func (i *Instance) Endpoint() Endpoint

Endpoint returns the instance address and its API key.

func (*Instance) ExitReason

func (i *Instance) ExitReason() error

ExitReason reports how the process ended; call it after Wait, it is nil while the process runs.

func (*Instance) Pid

func (i *Instance) Pid() int

Pid returns the child process id.

func (*Instance) Plan

func (i *Instance) Plan() *LaunchPlan

Plan returns the read-only plan retained by the instance.

func (*Instance) Stop

func (i *Instance) Stop(ctx context.Context) error

Stop terminates gracefully: SIGTERM, then SIGKILL when ctx expires.

func (*Instance) Wait

func (i *Instance) Wait() <-chan struct{}

Wait returns a channel closed when the process exits.

type Kind

type Kind = executor.Kind

Kind is the executor family (see pkg/engine/executor).

func ParseKind

func ParseKind(s string) (Kind, error)

ParseKind validates the exedra.engine value of a model.

func SupportedKinds

func SupportedKinds() []Kind

SupportedKinds lists executor families in stable order.

type LaunchPlan

type LaunchPlan struct {
	Spec    ModelSpec  `json:"spec"`
	Model   *ModelInfo `json:"model"`
	Command string     `json:"command"`
	Args    []string   `json:"args"`
	Env     []string   `json:"env,omitempty"`
	Host    string     `json:"host"`
	Port    int        `json:"port"`
	APIKey  string     `json:"api_key,omitempty"`
	Ready   string     `json:"ready"` // readiness HTTP path
}

LaunchPlan is a serializable launch decision. Args and Env may be edited before Launch; Host, Port and APIKey are read back from them, never set. Command, Spec, Model and Ready are the caller's to keep as planned. Launch retains the plan: do not modify it afterwards.

func Plan

func Plan(spec ModelSpec, hw *Hardware, c Constraints) (*LaunchPlan, error)

Plan layers RunArgs, engine flags, Overrides and caller extras; the merged argv, not the request, decides the endpoint.

func (*LaunchPlan) URL

func (p *LaunchPlan) URL() string

URL is the instance base address.

type ModelInfo

type ModelInfo struct {
	Path         string `json:"path"`
	Size         int64  `json:"size"`
	Architecture string `json:"architecture"`
	Name         string `json:"name"`
	Quant        string `json:"quant"`
	Blocks       int    `json:"blocks"`
	Context      int    `json:"context"`
}

ModelInfo is GGUF metadata relevant to planning.

func Inspect

func Inspect(path string) (*ModelInfo, error)

Inspect reads GGUF metadata without loading the weights.

type ModelSpec

type ModelSpec struct {
	ID           string   `json:"id"`                     // served name, "" = executor default
	Path         string   `json:"path"`                   // GGUF weights
	Kind         Kind     `json:"kind"`                   // executor family
	Capabilities []string `json:"capabilities,omitempty"` // llm | vlm | embeddings | rerank | asr
	RunArgs      []string `json:"run_args,omitempty"`     // model-recommended flags, lowest priority
}

ModelSpec describes one model to execute; path-based on purpose, the store stays unknown here.

type Override

type Override struct {
	Args []string          `yaml:"args" json:"args,omitempty"`
	Env  map[string]string `yaml:"env" json:"env,omitempty"`
}

Override is one launch customization layer.