backend

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 26, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package backend provides the interface and client for communicating with provisioning backends.

Fred supports multiple backends, each responsible for provisioning a specific type of resource (e.g., Kubernetes deployments, GPU allocations, VMs). The Router directs leases to the appropriate backend based on exact SKU UUID matching.

Backend Interface

Any backend must implement the Backend interface, which defines twelve operations:

  • Provision: Start async provisioning, backend calls callback URL when done
  • GetInfo: Retrieve connection details for a provisioned resource
  • GetProvision: Retrieve provision diagnostics (status, error, fail count)
  • GetLogs: Retrieve container logs for a provision
  • Deprovision: Clean up resources (must be idempotent)
  • ListProvisions: Return all provisioned resources (for reconciliation)
  • RefreshState: Synchronize in-memory state with actual resources
  • Restart: Restart containers without changing the manifest
  • Update: Deploy a new manifest, replacing containers
  • GetReleases: Return the release (deployment) history for a lease
  • Health: Report whether the backend can accept requests
  • Name: Return the backend's configured name

HTTPClient

The HTTPClient implements the Backend interface for HTTP-based backends. It includes:

  • Circuit breaker (sony/gobreaker) for fault tolerance
  • Configurable timeouts
  • Prometheus metrics for all operations

Router

The Router matches leases to backends using exact SKU UUIDs: 

backends:
  - name: docker-1
    skus:
      - "a1b2c3d4-e5f6-7890-abcd-1234567890ab"
  - name: default
    default: true          # Fallback for unmatched SKUs

Callback Protocol

After async provisioning completes, backends call fred's callback URL: 

POST {callback_url}
X-Fred-Signature: t=<unix-timestamp>,sha256=<hmac-sha256-hex>
Content-Type: application/json

{"lease_uuid": "...", "status": "success"|"failed", "error": "..."}

The HMAC is computed over the four-field canonical string "<timestamp>\n<METHOD>\n<canonical-URI>\n<hex(sha256(body))>" — binding the method and URI prevents cross-endpoint replay, and hashing the body makes the canonical string binary-safe. Callbacks older than 5 minutes are rejected (replay protection); timestamps up to 1 minute in the future are accepted (clock skew tolerance). See internal/hmacauth for the reference implementation.

Index

Constants

View Source 
const (
	DefaultMaxInfoBytes             int64 = 1 << 20  // 1 MiB — single lease info
	DefaultMaxProvisionBytes        int64 = 1 << 20  // 1 MiB — single provision record
	DefaultMaxProvisionsBytes       int64 = 8 << 20  // 8 MiB — list of all provisions
	DefaultMaxLookupProvisionsBytes int64 = 8 << 20  // 8 MiB — filtered provisions lookup; matches MaxProvisionsBytes because each ProvisionInfo carries an unbounded LastError and stack leases carry unbounded ServiceImages
	DefaultMaxLogsBytes             int64 = 16 << 20 // 16 MiB — container logs can be large
	DefaultMaxReleasesBytes         int64 = 8 << 20  // 8 MiB — release history with manifests
	DefaultMaxStatsBytes            int64 = 1 << 20  // 1 MiB — load stats snapshot (small JSON)
	DefaultMaxRetentionsBytes       int64 = 1 << 20  // 1 MiB — retained-lease list (small, UUID-only JSON)

	// DefaultProvisionsPageLimit is the page size the client requests on
	// GET /provisions. The server coerces a larger value down to MaxPageLimit.
	DefaultProvisionsPageLimit = 1000
)

Default response body size limits (defense-in-depth against buggy/misrouted backends).

View Source 
const MaxLookupUUIDs = 100

MaxLookupUUIDs caps the number of lease UUIDs accepted by a single LookupProvisions call (and the matching server-side filter on GET /provisions?lease_uuid=...). Shared across the HTTP client, the docker-backend handler, and the Fred-level GetWorkloads handler so a future change to the cap stays consistent end-to-end.

View Source 
const MaxPageLimit = 5000

MaxPageLimit is the server-side ceiling on a requested /provisions page size. A larger requested limit is coerced down to this value (AIP-158 "coerce down to the maximum") so a mis-set client limit cannot build an oversized page that trips the per-page byte guard and re-aborts the reconcile.

Variables

View Source 
var (
	ErrUnknownSKU      = fmt.Errorf("%w: unknown SKU", ErrValidation)
	ErrInvalidManifest = fmt.Errorf("%w: invalid manifest", ErrValidation)
	ErrImageNotAllowed = fmt.Errorf("%w: image not allowed", ErrValidation)
)

Validation sub-category sentinels. These wrap ErrValidation so errors.Is(err, ErrValidation) still works, while allowing callers to classify the failure without string matching.

View Source 
var ErrAlreadyProvisioned = errors.New("lease already provisioned")

ErrAlreadyProvisioned is returned when attempting to provision an already provisioned lease.

View Source 
var ErrCircuitOpen = errors.New("circuit breaker is open")

ErrCircuitOpen is returned when the circuit breaker is open.

View Source 
var ErrInsufficientResources = errors.New("insufficient resources")

ErrInsufficientResources is returned when there are not enough resources to fulfill a provision request.

View Source 
var ErrInvalidState = errors.New("invalid state for operation")

ErrInvalidState is returned when an operation is not valid for the current lease state.

View Source 
var ErrNotProvisioned = errors.New("lease not provisioned")

ErrNotProvisioned is returned when a lease is not yet provisioned.

View Source 
var ErrNotRetained = errors.New("no retained data for lease")

ErrNotRetained is returned when no retained data exists for the original lease (absent, expired, or owned by a different tenant). Distinct from ErrNotProvisioned (which concerns live provisions).

View Source 
var ErrResponseTooLarge = errors.New("response body too large")

ErrResponseTooLarge is returned when a backend response body exceeds the configured size limit.

View Source 
var ErrValidation = errors.New("validation error")

ErrValidation is returned when a provision request fails pre-flight validation (e.g., unknown SKU, invalid manifest, disallowed image registry).

Functions

func NormalizeProvisionRequest 

func NormalizeProvisionRequest(req *ProvisionRequest) error

NormalizeProvisionRequest applies boundary normalization to a provision request: any single-item request that lacks ServiceName is auto-tagged with defaultServiceName, transitioning legacy wire-format leases into the stack-shaped contract every downstream component now expects. Named items are passed through. Mixed-presence and multi-unnamed combinations are rejected — both were structurally invalid under the legacy contract too, so this only formalizes the rejection.

The normalization is in-place; callers should invoke this at the very entry of Provision/Update, before any docker work or pool allocation, so the rest of the path operates on uniformly stack-shaped state.

func ParseProvisionsPageParams added in v0.5.0 

func ParseProvisionsPageParams(q url.Values) (limit int, continueToken string, err error)

ParseProvisionsPageParams extracts pagination params from a /provisions query. It returns a non-nil error (which handlers map to HTTP 400) when 'limit' is present but not a non-negative integer, or 'continue' is present but not a valid UUID. Absent params yield (0, "", nil) — the unpaginated passthrough.

Types

type Backend 

type Backend interface {
	// Provision starts async provisioning of a resource.
	// The backend will call the callback URL when provisioning completes.
	Provision(ctx context.Context, req ProvisionRequest) error

	// GetInfo returns lease information including connection details.
	// Returns ErrNotProvisioned if the lease is not yet provisioned.
	GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

	// Deprovision releases resources for a lease. Must be idempotent.
	Deprovision(ctx context.Context, leaseUUID string) error

	// ListProvisions returns all currently provisioned resources.
	// Used by the reconciler for orphan detection.
	ListProvisions(ctx context.Context) ([]ProvisionInfo, error)

	// LookupProvisions returns provision info for the requested lease UUIDs.
	// Missing leases are simply absent from the returned slice (not an error).
	// Used by the /workloads endpoint to fetch metadata for a known set of leases
	// without scanning the full provisions corpus. Caller is responsible for
	// enforcing MaxLookupUUIDs; implementations may also enforce it defensively.
	//
	// Named LookupProvisions (not GetProvisions) to avoid visual ambiguity with
	// the existing singular GetProvision.
	LookupProvisions(ctx context.Context, leaseUUIDs []string) ([]ProvisionInfo, error)

	// Health checks if the backend is reachable and healthy.
	// Returns nil if healthy, error otherwise.
	Health(ctx context.Context) error

	// RefreshState synchronizes in-memory provision state with the
	// underlying infrastructure. Backends should query the real
	// container/VM state and update their internal tracking.
	// Called by the reconciler before ListProvisions to avoid stale reads.
	RefreshState(ctx context.Context) error

	// GetProvision returns status information for a single provision.
	// Returns ErrNotProvisioned if the lease is not found.
	GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

	// GetLogs returns container logs for a provisioned lease.
	// The tail parameter limits the number of log lines per container.
	// Returns ErrNotProvisioned if the lease is not found.
	GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

	// Restart restarts containers for a lease without changing the manifest.
	// Returns ErrNotProvisioned if lease not found, ErrInvalidState if not restartable.
	Restart(ctx context.Context, req RestartRequest) error

	// Update deploys a new manifest for a lease, replacing containers.
	// Returns ErrNotProvisioned if lease not found, ErrInvalidState if not updatable.
	Update(ctx context.Context, req UpdateRequest) error

	// Restore re-provisions a new lease from the retained volumes of a prior
	// soft-deleted lease (ENG-325). Returns ErrNotRetained if no retained data
	// exists for FromLeaseUUID, ErrInvalidState if the target lease is not in
	// a valid state for restore, ErrValidation for bad inputs.
	Restore(ctx context.Context, req RestoreRequest) error

	// ReconcileCustomDomain reapplies the per-LeaseItem custom_domain values
	// from chain onto the running provision. When any item's CustomDomain
	// differs from the in-memory provision state, the backend snapshots the
	// current values, updates them, and triggers a Restart() to re-emit
	// Traefik labels. On Restart failure the in-memory state is rolled back
	// so the next reconciler tick can retry.
	//
	// No-op when:
	//   - the lease is not provisioned by this backend (silent),
	//   - the provision is not currently active,
	//   - all items already match the incoming values.
	//
	// Idempotent: repeated calls with the same items list produce no
	// container churn beyond the first reconciling call.
	ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) error

	// GetReleases returns the release history for a lease.
	// Returns ErrNotProvisioned if lease not found.
	GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

	// GetLoadStats returns the backend's current resource-load snapshot,
	// used for least-loaded provision routing. Implementations that do not
	// track load may return a snapshot whose CPUAllocatedRatio is not ok.
	GetLoadStats(ctx context.Context) (*LoadStats, error)

	// ListRetentions returns the leases whose data this backend currently
	// retains (soft-deleted, awaiting restore or grace-reap). Used by the
	// reconciler to keep placement affinity for retained leases (ENG-333).
	// Backends without retention (e.g. k3s) return an empty slice.
	ListRetentions(ctx context.Context) ([]RetainedLease, error)

	// Name returns the backend's configured name.
	Name() string
}

Backend defines the interface for interacting with a provisioning backend. Any backend (Kubernetes, GPU, VM, etc.) must implement these operations.

type BackendEntry 

type BackendEntry struct {
	Backend   Backend
	Match     MatchCriteria
	IsDefault bool
}

BackendEntry pairs a backend with its matching criteria.

type BackendHealth 

type BackendHealth struct {
	Name    string `json:"name"`
	Healthy bool   `json:"healthy"`
	Error   string `json:"error,omitempty"`
}

BackendHealth represents the health status of a single backend.

type CallbackPayload 

type CallbackPayload struct {
	LeaseUUID string         `json:"lease_uuid"`
	Status    CallbackStatus `json:"status"` // "success", "failed", or "deprovisioned"
	Error     string         `json:"error,omitempty"`
	Backend   string         `json:"backend,omitempty"` // Backend name; empty from pre-upgrade senders.
	// Retained is set true on a deprovisioned callback when the backend actually
	// soft-deleted (retained) the lease's volumes. Best-effort ground truth for
	// the optimistic push; the queryable retention status is the durable backstop.
	Retained bool `json:"retained,omitempty"`
}

CallbackPayload is sent by backends to fred's callback endpoint.

type CallbackStatus 

type CallbackStatus string

CallbackStatus represents the status sent in a callback payload. 

const (
	CallbackStatusSuccess       CallbackStatus = "success"
	CallbackStatusFailed        CallbackStatus = "failed"
	CallbackStatusDeprovisioned CallbackStatus = "deprovisioned"
)

Callback status constants.

type HTTPClient 

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient implements Backend using HTTP calls to a backend service.

func NewHTTPClient 

func NewHTTPClient(cfg HTTPClientConfig) *HTTPClient

NewHTTPClient creates a new HTTP backend client.

func (*HTTPClient) Deprovision 

func (c *HTTPClient) Deprovision(ctx context.Context, leaseUUID string) (err error)

Deprovision releases resources for a lease.

func (*HTTPClient) GetInfo 

func (c *HTTPClient) GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

GetInfo retrieves lease information including connection details.

func (*HTTPClient) GetLoadStats added in v0.5.0 

func (c *HTTPClient) GetLoadStats(ctx context.Context) (*LoadStats, error)

GetLoadStats retrieves the backend's current resource-load snapshot from GET /stats. Used by the router for least-loaded provision placement.

func (*HTTPClient) GetLogs 

func (c *HTTPClient) GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

GetLogs retrieves container logs for a provisioned lease.

func (*HTTPClient) GetProvision 

func (c *HTTPClient) GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

GetProvision retrieves status information for a single provision.

func (*HTTPClient) GetReleases 

func (c *HTTPClient) GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

GetReleases retrieves release history for a lease.

func (*HTTPClient) Health 

func (c *HTTPClient) Health(ctx context.Context) error

Health checks if the backend is reachable and healthy. It sends a GET request to /health on the backend.

func (*HTTPClient) ListProvisions 

func (c *HTTPClient) ListProvisions(ctx context.Context) (_ []ProvisionInfo, err error)

ListProvisions returns all provisioned resources from this backend. It walks the keyset-paginated /provisions endpoint, reassembling the complete set; it returns an error rather than a partial set (complete-or-error) so the reconciler never acts on incomplete data.

func (*HTTPClient) ListRetentions added in v0.5.0 

func (c *HTTPClient) ListRetentions(ctx context.Context) ([]RetainedLease, error)

ListRetentions retrieves the leases whose data this backend currently retains (GET /retentions). Used by the reconciler for restore backend affinity.

func (*HTTPClient) LookupProvisions 

func (c *HTTPClient) LookupProvisions(ctx context.Context, uuids []string) ([]ProvisionInfo, error)

LookupProvisions returns provision info for the requested lease UUIDs. Missing leases are absent from the returned slice (not an error).

Wire path: GET {baseURL}/provisions?lease_uuid=...&lease_uuid=... The docker-backend handler shares the same route as the unfiltered ListProvisions path; the filter param toggles the subset behavior. Same wire type either way.

func (*HTTPClient) Name 

func (c *HTTPClient) Name() string

Name returns the backend's configured name.

func (*HTTPClient) Provision 

func (c *HTTPClient) Provision(ctx context.Context, req ProvisionRequest) (err error)

Provision sends a provision request to the backend.

func (*HTTPClient) ReconcileCustomDomain 

func (c *HTTPClient) ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) (err error)

ReconcileCustomDomain forwards a reconciliation request to the remote backend. The reconciler calls this on every active lease each tick; the remote side decides whether any action is needed (idempotent on no-change).

func (*HTTPClient) RefreshState 

func (c *HTTPClient) RefreshState(ctx context.Context) error

RefreshState is a no-op for remote backends (they refresh server-side).

func (*HTTPClient) Restart 

func (c *HTTPClient) Restart(ctx context.Context, req RestartRequest) (err error)

Restart sends a restart request to the backend.

func (*HTTPClient) Restore added in v0.5.0 

func (c *HTTPClient) Restore(ctx context.Context, req RestoreRequest) (err error)

Restore sends a restore request to the backend.

func (*HTTPClient) Update 

func (c *HTTPClient) Update(ctx context.Context, req UpdateRequest) (err error)

Update sends an update request to the backend.

type HTTPClientConfig 

type HTTPClientConfig struct {
	Name                string
	BaseURL             string
	Timeout             time.Duration
	MaxIdleConns        int // Max idle connections across all hosts (default: 100)
	MaxIdleConnsPerHost int // Max idle connections per host (default: 10)
	Secret              string

	// TLSClientConfig, when non-nil, is applied to the backend HTTP transport
	// (private-CA trust and/or a client certificate for mTLS). Built by the
	// caller from per-backend config so this package performs no file I/O.
	TLSClientConfig *tls.Config

	// Circuit breaker settings
	CBMaxRequests   uint32        // Max requests in half-open state (default: 1)
	CBInterval      time.Duration // Interval to clear counts in closed state (default: 0, never clear)
	CBTimeout       time.Duration // Time to wait before transitioning from open to half-open (default: 60s)
	CBFailureThresh uint32        // Number of failures to trip the breaker (default: 5)

	// Response body size limits (0 = use default). Defense-in-depth caps to prevent
	// a buggy or misrouted backend from pressuring memory with unbounded responses.
	MaxInfoBytes             int64 // GetInfo response limit (default: 1 MiB)
	MaxProvisionBytes        int64 // GetProvision response limit (default: 1 MiB)
	MaxProvisionsBytes       int64 // ListProvisions response limit (default: 8 MiB)
	MaxLookupProvisionsBytes int64 // LookupProvisions response limit (default: 8 MiB)
	MaxLogsBytes             int64 // GetLogs response limit (default: 16 MiB)
	MaxReleasesBytes         int64 // GetReleases response limit (default: 8 MiB)
	MaxStatsBytes            int64 // GetLoadStats response limit (default: 1 MiB)
	MaxRetentionsBytes       int64 // ListRetentions response limit (default: 1 MiB)
	ProvisionsPageLimit      int   // /provisions page size requested by the client (default: 1000)

	// Optional Prometheus metrics. When nil, metric recording is skipped.
	// This prevents binaries that don't use these metrics (e.g., docker-backend)
	// from registering phantom fred-level metrics via transitive imports.
	RequestDuration     *prometheus.HistogramVec // labels: backend, operation, status
	RequestsTotal       *prometheus.CounterVec   // labels: backend, operation, status
	CircuitBreakerState *prometheus.GaugeVec     // labels: backend
}

HTTPClientConfig configures an HTTP backend client.

type LeaseInfo 

type LeaseInfo struct {
	Host      string                  `json:"host,omitempty"`
	FQDN      string                  `json:"fqdn,omitempty"`
	Protocol  string                  `json:"protocol,omitempty"`
	Ports     map[string]PortBinding  `json:"ports,omitempty"`
	Instances []LeaseInstance         `json:"instances,omitempty"`
	Services  map[string]LeaseService `json:"services,omitempty"`
	Metadata  map[string]string       `json:"metadata,omitempty"`
}

LeaseInfo contains backend-specific information about a provisioned lease.

type LeaseInstance 

type LeaseInstance struct {
	InstanceIndex int                    `json:"instance_index"`
	ContainerID   string                 `json:"container_id,omitempty"`
	Image         string                 `json:"image,omitempty"`
	Status        string                 `json:"status,omitempty"`
	FQDN          string                 `json:"fqdn,omitempty"`
	Ports         map[string]PortBinding `json:"ports,omitempty"`
}

LeaseInstance represents a single provisioned instance (container/pod).

type LeaseItem 

type LeaseItem struct {
	SKU         string `json:"sku"`
	Quantity    int    `json:"quantity"`
	ServiceName string `json:"service_name,omitempty"`
	// CustomDomain is the optional FQDN the tenant has assigned to this item.
	// When non-empty (and the service has a routable HTTP port), the backend
	// emits a secondary Traefik router on the item's container(s) routing
	// Host(<CustomDomain>) to the same loadbalancer service. Validated on-chain
	// in MsgSetItemCustomDomain; Fred re-runs cheap defense-in-depth checks
	// before applying labels.
	CustomDomain string `json:"custom_domain,omitempty"`
}

LeaseItem represents a single SKU with its quantity in a lease.

type LeaseService 

type LeaseService struct {
	FQDN      string          `json:"fqdn,omitempty"`
	Instances []LeaseInstance `json:"instances,omitempty"`
}

LeaseService groups instances belonging to a single service in a stack lease.

type LeaseStatusEvent 

type LeaseStatusEvent struct {
	LeaseUUID string          `json:"lease_uuid"`
	Status    ProvisionStatus `json:"status"`
	Error     string          `json:"error,omitempty"`
	Timestamp time.Time       `json:"timestamp"`
}

LeaseStatusEvent is published when a lease's provisioning status changes. Used for real-time delivery (e.g., WebSocket) to notify tenants of status transitions.

type ListProvisionsResponse 

type ListProvisionsResponse struct {
	Provisions []ProvisionInfo `json:"provisions"`
	// Continue is the keyset cursor for the next page: the LeaseUUID of the last
	// item in this page when more remain, or "" when the set is exhausted.
	// Absent (omitempty) on the unpaginated path and from pre-pagination servers.
	Continue string `json:"continue,omitempty"`
}

ListProvisionsResponse is the response from the /provisions endpoint.

type ListRetentionsResponse added in v0.5.0 

type ListRetentionsResponse struct {
	Retentions []RetainedLease `json:"retentions"`
}

ListRetentionsResponse is the response from the GET /retentions endpoint.

type LoadStats added in v0.5.0 

type LoadStats struct {
	TotalCPUCores     float64 `json:"total_cpu_cores"`
	AllocatedCPUCores float64 `json:"allocated_cpu_cores"`
	ActiveContainers  int     `json:"active_containers"`
}

LoadStats is the backend load snapshot fred consumes for least-loaded provision routing. It is decoded from the backend's GET /stats response. Only the CPU-related fields are needed for routing — CPU is the binding resource on current configurations, so memory/disk are intentionally ignored. Unknown JSON fields in the response are discarded by the decoder.

func (*LoadStats) CPUAllocatedRatio added in v0.5.0 

func (s *LoadStats) CPUAllocatedRatio() (ratio float64, ok bool)

CPUAllocatedRatio returns the fraction of CPU allocated (allocated/total) and ok=true when that ratio is meaningful. ok is false when total capacity is unknown or non-positive (e.g., a backend that does not report CPU capacity, or a nil snapshot), signaling the caller to treat this backend as having no usable load signal.

type MatchCriteria 

type MatchCriteria struct {
	SKUs []string // Match if SKU is in this exact list
}

MatchCriteria defines how to match a lease to a backend.

type MockBackend 

type MockBackend struct {
	// contains filtered or unexported fields
}

MockBackend is an in-memory backend for testing. It simulates provisioning with configurable delays.

func NewMockBackend 

func NewMockBackend(cfg MockBackendConfig) *MockBackend

NewMockBackend creates a new mock backend for testing.

func (*MockBackend) Clear 

func (m *MockBackend) Clear()

Clear removes all provisions (for testing).

func (*MockBackend) Deprovision 

func (m *MockBackend) Deprovision(ctx context.Context, leaseUUID string) error

Deprovision removes a provision.

func (*MockBackend) GetInfo 

func (m *MockBackend) GetInfo(ctx context.Context, leaseUUID string) (*LeaseInfo, error)

GetInfo returns mock lease information including connection details.

func (*MockBackend) GetLoadStats added in v0.5.0 

func (m *MockBackend) GetLoadStats(_ context.Context) (*LoadStats, error)

GetLoadStats returns the configured load snapshot (a copy, so callers cannot mutate mock state), or (nil, nil) when unset.

func (*MockBackend) GetLogs 

func (m *MockBackend) GetLogs(ctx context.Context, leaseUUID string, tail int) (map[string]string, error)

GetLogs returns empty logs for mock backend (Backend interface).

func (*MockBackend) GetMockProvision 

func (m *MockBackend) GetMockProvision(leaseUUID string) (*mockProvision, bool)

GetMockProvision returns the internal mock provision (for testing).

func (*MockBackend) GetProvision 

func (m *MockBackend) GetProvision(ctx context.Context, leaseUUID string) (*ProvisionInfo, error)

GetProvision returns status information for a single provision (Backend interface).

func (*MockBackend) GetReleases 

func (m *MockBackend) GetReleases(ctx context.Context, leaseUUID string) ([]ReleaseInfo, error)

GetReleases returns empty releases for mock backend.

func (*MockBackend) Health 

func (m *MockBackend) Health(ctx context.Context) error

Health always returns nil (healthy) for mock backend.

func (*MockBackend) ListProvisions 

func (m *MockBackend) ListProvisions(ctx context.Context) ([]ProvisionInfo, error)

ListProvisions returns all provisions.

func (*MockBackend) ListRetentions added in v0.5.0 

func (m *MockBackend) ListRetentions(_ context.Context) ([]RetainedLease, error)

ListRetentions returns the configured retained leases (a copy), or an empty non-nil slice when unset.

func (*MockBackend) LookupProvisions 

func (m *MockBackend) LookupProvisions(ctx context.Context, uuids []string) ([]ProvisionInfo, error)

LookupProvisions returns provision info for the requested lease UUIDs. Missing leases are absent from the returned slice.

func (*MockBackend) Name 

func (m *MockBackend) Name() string

Name returns the backend's name.

func (*MockBackend) Provision 

func (m *MockBackend) Provision(ctx context.Context, req ProvisionRequest) error

Provision simulates starting a provision operation.

func (*MockBackend) ReconcileCustomDomain 

func (m *MockBackend) ReconcileCustomDomain(ctx context.Context, leaseUUID string, items []LeaseItem) error

ReconcileCustomDomain is a no-op for mock backend.

func (*MockBackend) RefreshState 

func (m *MockBackend) RefreshState(ctx context.Context) error

RefreshState is a no-op for mock backend.

func (*MockBackend) Restart 

func (m *MockBackend) Restart(ctx context.Context, req RestartRequest) error

Restart is a no-op for mock backend.

func (*MockBackend) Restore added in v0.5.0 

func (m *MockBackend) Restore(ctx context.Context, req RestoreRequest) error

Restore is a no-op for mock backend — returns ErrNotRetained unless the source lease exists in the provisions map (simulating "has retained data").

func (*MockBackend) SetCallbackFunc 

func (m *MockBackend) SetCallbackFunc(fn func(CallbackPayload))

SetCallbackFunc sets the function to call when provisioning completes. This simulates the backend calling fred's callback endpoint.

func (*MockBackend) SetLoadStats added in v0.5.0 

func (m *MockBackend) SetLoadStats(stats *LoadStats)

SetLoadStats configures the snapshot returned by GetLoadStats. Pass nil to simulate a backend with no usable load signal.

func (*MockBackend) SetProvisionStatus 

func (m *MockBackend) SetProvisionStatus(leaseUUID string, status ProvisionStatus)

SetProvisionStatus manually sets a provision's status (for testing).

func (*MockBackend) SetRetentions added in v0.5.0 

func (m *MockBackend) SetRetentions(r []RetainedLease)

SetRetentions configures the slice returned by ListRetentions.

func (*MockBackend) Update 

func (m *MockBackend) Update(ctx context.Context, req UpdateRequest) error

Update is a no-op for mock backend.

type MockBackendConfig 

type MockBackendConfig struct {
	Name           string
	ProvisionDelay time.Duration // Simulated provisioning time
}

MockBackendConfig configures a mock backend.

type PortBinding 

type PortBinding struct {
	HostIP   string `json:"host_ip"`
	HostPort string `json:"host_port"`
}

PortBinding represents a port mapping from container to host as reported by a backend.

type ProvisionInfo 

type ProvisionInfo struct {
	LeaseUUID    string          `json:"lease_uuid"`
	ProviderUUID string          `json:"provider_uuid"`
	Status       ProvisionStatus `json:"status"` // see ProvisionStatus* constants
	CreatedAt    time.Time       `json:"created_at"`
	FailCount    int             `json:"fail_count"`
	LastError    string          `json:"last_error,omitempty"`
	BackendName  string          `json:"-"` // Set by the backend or reconciler; excluded from JSON serialization

	// RetainedUntil is the grace-window deadline (CreatedAt + RetentionMaxAge)
	// for a soft-deleted (Status=retained) lease. Zero for live provisions.
	RetainedUntil time.Time `json:"retained_until,omitempty"`
	// Tenant is the owning tenant. It crosses the backend→providerd hop (an
	// HMAC-signed, trusted internal hop, like RestoreRequest.Tenant) so the
	// closed-lease authz fallback (when the chain has pruned the lease) can bind
	// a retained record to its owner. It MUST NOT be copied into tenant-facing
	// API responses (LeaseStatusResponse/LeaseProvisionResponse), which would
	// leak one tenant's address to another.
	Tenant string `json:"tenant,omitempty"`

	// Workload metadata — populated by ListProvisions/GetProvision to describe what is running.
	Image         string            `json:"image,omitempty"`          // Docker image (non-stack leases)
	SKU           string            `json:"sku,omitempty"`            // SKU identifier (non-stack leases)
	Quantity      int               `json:"quantity"`                 // Total expected container count
	Items         []LeaseItem       `json:"items,omitempty"`          // Per-service items (stack leases)
	ServiceImages map[string]string `json:"service_images,omitempty"` // service name → image (stack leases)
}

ProvisionInfo describes a single provisioned resource.

func PaginateProvisions added in v0.5.0 

func PaginateProvisions(all []ProvisionInfo, continueToken string, limit int) (page []ProvisionInfo, next string)

PaginateProvisions returns one keyset page of all, sorted by LeaseUUID ascending, containing the entries strictly greater than continueToken.

  • limit <= 0 -> returns (all, sorted) with next "" (unpaginated passthrough)
  • limit > MaxPageLimit -> limit is coerced down to MaxPageLimit

next is the LeaseUUID of the last returned element iff a full page was returned AND more elements remain after it; otherwise "" (the set is exhausted). next is always strictly greater than continueToken, so a client may use a strict-increase loop guard without page-boundary false positives. The returned page is always non-nil so it serializes as [] rather than null.

Precondition: LeaseUUID is a unique total order (lease identity is a UUIDv7). Keyset paging silently skips or duplicates at page boundaries if the sort key is non-unique; if the sort key ever becomes composite, switch the cursor to an encoded versioned tuple per design spec §4.

type ProvisionRequest 

type ProvisionRequest struct {
	LeaseUUID    string      `json:"lease_uuid"`
	Tenant       string      `json:"tenant"`
	ProviderUUID string      `json:"provider_uuid"`
	Items        []LeaseItem `json:"items"`
	CallbackURL  string      `json:"callback_url"`
	Payload      []byte      `json:"payload,omitempty"`
	PayloadHash  string      `json:"payload_hash,omitempty"`
}

ProvisionRequest contains the data needed to provision a resource.

func (ProvisionRequest) RoutingSKU 

func (r ProvisionRequest) RoutingSKU() string

RoutingSKU returns the SKU of the first item for backend routing decisions.

Why this exists: A lease may contain multiple items with different SKUs (e.g., [{sku: "docker-micro", qty: 2}, {sku: "docker-large", qty: 1}]). However, all items in a single lease are guaranteed to belong to the same provider - this is enforced by the chain. Therefore, any SKU from the lease can be used to determine which backend should handle the request.

This method returns the first SKU purely for routing. It should NOT be used to determine resource allocation - use Items directly for that.

func (ProvisionRequest) TotalQuantity 

func (r ProvisionRequest) TotalQuantity() int

TotalQuantity returns the sum of quantities across all items.

type ProvisionResponse 

type ProvisionResponse struct {
	ProvisionID string `json:"provision_id"`
}

ProvisionResponse is returned by the backend after accepting a provision request.

type ProvisionStatus 

type ProvisionStatus string

ProvisionStatus represents the status of a provisioned resource. 

const (
	ProvisionStatusProvisioning   ProvisionStatus = "provisioning"
	ProvisionStatusReady          ProvisionStatus = "ready"
	ProvisionStatusFailing        ProvisionStatus = "failing"
	ProvisionStatusFailed         ProvisionStatus = "failed"
	ProvisionStatusUnknown        ProvisionStatus = "unknown"
	ProvisionStatusRestarting     ProvisionStatus = "restarting"
	ProvisionStatusUpdating       ProvisionStatus = "updating"
	ProvisionStatusDeprovisioning ProvisionStatus = "deprovisioning"
	// ProvisionStatusRetained is published to the tenant at lease close/expire
	// time to signal that managed volumes may have been kept under a
	// fred-retained- namespace if the backend has retain_on_close enabled.
	// This is a best-effort hint; the backend may not have retention configured.
	ProvisionStatusRetained ProvisionStatus = "retained"
)

Provision status constants.

ProvisionStatusFailing marks the window between a container's death being detected and the terminal Failed callback being emitted. A Deprovision request arriving in this window transitions the lease straight to Deprovisioning without ever reaching Failed — preventing a stale Failed callback under concurrent ownership transfer.

type ReconcileCustomDomainRequest 

type ReconcileCustomDomainRequest struct {
	LeaseUUID string      `json:"lease_uuid"`
	Items     []LeaseItem `json:"items"`
}

ReconcileCustomDomainRequest is the wire format for POST /reconcile_custom_domain.

type ReleaseInfo 

type ReleaseInfo struct {
	Version   int       `json:"version"`
	Image     string    `json:"image"`
	Status    string    `json:"status"`
	CreatedAt time.Time `json:"created_at"`
	Error     string    `json:"error,omitempty"`
	Manifest  []byte    `json:"manifest"`
}

ReleaseInfo describes a single release in the history.

type RestartRequest 

type RestartRequest struct {
	LeaseUUID   string `json:"lease_uuid"`
	CallbackURL string `json:"callback_url"`
}

RestartRequest contains the data needed to restart a lease's containers.

type RestoreRequest added in v0.5.0 

type RestoreRequest struct {
	LeaseUUID     string      `json:"lease_uuid"`      // NEW lease
	FromLeaseUUID string      `json:"from_lease_uuid"` // original (retained) lease
	Tenant        string      `json:"tenant"`
	ProviderUUID  string      `json:"provider_uuid"` // when non-empty, cross-checked against the retained record
	Items         []LeaseItem `json:"items"`         // must match the retained set
	CallbackURL   string      `json:"callback_url"`
}

RestoreRequest contains the data needed to restore a soft-deleted lease's retained volumes into a NEW lease (ENG-325). FromLeaseUUID identifies the original (retained) lease whose data is adopted; LeaseUUID is the new lease the data is restored into. Items must match the retained set's shape (service-name → summed-quantity).

type RetainedLease added in v0.5.0 

type RetainedLease struct {
	LeaseUUID string `json:"lease_uuid"`
}

RetainedLease identifies a lease whose data this backend currently retains (soft-deleted, awaiting restore or grace-reap). The reconciler consumes this to keep placement affinity for retained leases so a restore routes to the backend holding the source data (ENG-333).

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Router routes requests to backends based on SKU matching.

func NewRouter 

func NewRouter(cfg RouterConfig) (*Router, error)

NewRouter creates a new backend router.

func (*Router) Backends 

func (r *Router) Backends() []Backend

Backends returns all unique backends for operations like reconciliation and health checks. The same backend may be registered multiple times with different SKU lists, but this method returns each backend only once (deduplicated by name).

func (*Router) Default 

func (r *Router) Default() Backend

Default returns the default backend.

func (*Router) GetBackendByName 

func (r *Router) GetBackendByName(name string) Backend

GetBackendByName returns a backend by its name. Returns nil if not found.

func (*Router) HealthCheck 

func (r *Router) HealthCheck(ctx context.Context) ([]BackendHealth, bool)

HealthCheck checks the health of all configured backends. Returns a slice of health statuses and an overall healthy flag.

func (*Router) Route 

func (r *Router) Route(sku string) Backend

Route returns the appropriate backend for the given SKU.

func (*Router) RouteAll 

func (r *Router) RouteAll(sku string) []Backend

RouteAll returns all backends that match the given SKU, deduplicated by name. If no backends match, returns nil.

func (*Router) RouteForProvision added in v0.5.0 

func (r *Router) RouteForProvision(ctx context.Context, sku string, inFlightByBackend map[string]int) Backend

RouteForProvision selects the SKU-matching backend with the lowest observed allocated-CPU ratio for a new provision. Ties (equal ratio within cpuRatioEpsilon — the burst case, where concurrent provisions read the same /stats snapshot) are broken by the fewest in-flight provisions, then by the round-robin counter so a burst of identical-state provisions spreads across the tied backends. Falls back to round-robin when no SKU-matching backend exposes usable load stats. inFlightByBackend may be nil (treated as all-zero).

This is a deliberate live-query design — each candidate's /stats is fetched at decision time — NOT a passive-counter load balancer (Envoy/NGINX style). The binding CPU-allocation signal lives on the backends and fred holds no local per-SKU CPU weights, so the only way to learn load is to ask. A GetLoadStats error or timeout demotes that candidate to "no usable signal" (excluded from the comparison; the error is recorded by doGet as fred_backend_requests_total{operation="get_load_stats",status="error"}). A residual herd window remains when concurrent provisions read the same pre-update /stats snapshot and the in-flight tiebreak does not separate them; it is tolerated because the backend's 503 admission gate hard-caps any over-targeted backend and provision QPS is low. The round-robin counter is shared with RouteRoundRobin, so tie rotation is intentionally approximate.

func (*Router) RouteRoundRobin 

func (r *Router) RouteRoundRobin(sku string) Backend

RouteRoundRobin distributes requests across all backends matching the SKU using round-robin selection. Falls back to the default backend if no match.

type RouterConfig 

type RouterConfig struct {
	Backends []BackendEntry

	// Optional Prometheus gauge for backend health status. When nil, HealthCheck
	// skips metric recording. This prevents binaries that don't use these metrics
	// from registering phantom fred-level gauges via transitive imports.
	BackendHealthy *prometheus.GaugeVec // labels: backend

	// AllocatedCPURatio, when non-nil, records the allocated-CPU ratio observed
	// for each backend at provision-routing time. Labels: backend.
	AllocatedCPURatio *prometheus.GaugeVec

	// RoutingFallback, when non-nil, counts provision-routing decisions that
	// fell back to round-robin because no candidate exposed usable load stats.
	RoutingFallback prometheus.Counter
}

RouterConfig configures the backend router.

type UpdateRequest 

type UpdateRequest struct {
	LeaseUUID   string `json:"lease_uuid"`
	CallbackURL string `json:"callback_url"`
	Payload     []byte `json:"payload"`
	PayloadHash string `json:"payload_hash,omitempty"`
}

UpdateRequest contains the data needed to update a lease to a new manifest.

type ValidationCode 

type ValidationCode string

ValidationCode identifies the sub-category of a validation error in HTTP responses. Backends include this in 400 JSON bodies so the client can reconstruct the correct sentinel error across the HTTP boundary. 

const (
	ValidationCodeUnknownSKU      ValidationCode = "unknown_sku"
	ValidationCodeInvalidManifest ValidationCode = "invalid_manifest"
	ValidationCodeImageNotAllowed ValidationCode = "image_not_allowed"
)

Validation code constants used in HTTP error responses.

func ClassifyValidationError 

func ClassifyValidationError(err error) ValidationCode

ClassifyValidationError returns the ValidationCode for a validation error. Returns "" if the error does not match any known sub-category.

Source Files

View all Source files

Directories

Path Synopsis
Package docker implements a Docker backend for Fred that provisions ephemeral containers with SKU-based resource profiles, registry allowlisting, and port mapping for tenant connectivity.
 Package docker implements a Docker backend for Fred that provisions ephemeral containers with SKU-based resource profiles, registry allowlisting, and port mapping for tenant connectivity.
Package k3s is the scaffold for Fred's K3s backend.
 Package k3s is the scaffold for Fred's K3s backend.
Package shared provides backend-agnostic components that can be reused across different backend implementations (Docker, Kubernetes, Nomad, etc.).
 Package shared provides backend-agnostic components that can be reused across different backend implementations (Docker, Kubernetes, Nomad, etc.).
leasesm
Package leasesm owns the per-lease state machine and actor, plus the substrate-agnostic seams they consume.
 Package leasesm owns the per-lease state machine and actor, plus the substrate-agnostic seams they consume.
manifest
Package manifest defines the tenant-facing container manifest schema (StackManifest, holding a map of named per-service Manifest entries), JSON parsing, and validation.
 Package manifest defines the tenant-facing container manifest schema (StackManifest, holding a map of named per-service Manifest entries), JSON parsing, and validation.
workbarrier
Package workbarrier provides a reference-counted barrier whose zero signal is exposed as a real channel, so callers can wait-with-timeout using a plain select without spawning helper goroutines that would leak when the count never drops to zero.
 Package workbarrier provides a reference-counted barrier whose zero signal is exposed as a real channel, so callers can wait-with-timeout using a plain select without spawning helper goroutines that would leak when the count never drops to zero.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.