capacity

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

capacity

A small, clean cloud capacity-provider interface for Go, plus a complete AWS Spot reference implementation and a conformance suite. It's the extension point for provisioning ephemeral compute (GPU workers) across clouds: implement one interface, pass the conformance suite, and a scheduler can shop and launch on your source of capacity.

The interface

type Provider interface {
    Name() string
    Quote(ctx, Requirement) ([]Offer, error)                      // live offers that satisfy the requirement
    Launch(ctx, id string, o Offer, spec WorkerSpec) (WorkerHandle, error) // provision a worker onto an offer
    Terminate(ctx, WorkerHandle) error                            // tear it down
    ListOwned(ctx) ([]OwnedInstance, error)                       // every instance we own — the anti-orphan safety net
}

That's the whole contract. Quote surfaces priced options; the caller scores them (this package is deliberately not opinionated about which offer to pick). ListOwned is the safety net: cross-check it against your own registry and terminate anything untracked, so no instance can bill forever after a state loss.

What's in the box

  • capacity — the Provider interface, domain types (Offer, Requirement, WorkerSpec, WorkerHandle), a fake provider for tests, and composable wrappers: gated (respect an operator on/off switch), multiregion, and multicloud (fan Quote across several providers).
  • capacity/aws — a working AWS EC2 Spot provider: quotes spot offers, launches tagged instances, terminates, and lists owned instances by tag.
  • InstanceAuditor — a small optional capability, separate from Provider: list every running instance in the account, owned or not. ListOwned only sees what you tagged, so it can never surface a leak from outside the system (an untagged image builder, a box someone forgot); an account-wide sweep can. Implemented by the AWS provider, and fanned across every region/cloud by multiregion/multicloud — providers that don't implement it are skipped rather than failing the sweep.
  • Conformance suite — capacity.RunConformance(t, newProvider, …) exercises any implementation against the contract, so "I added a provider" means "I added a conformant provider."

Install

go get github.com/getotium/capacity

Adding a provider

Implement Provider for your cloud (GCP, Azure, OCI, RunPod, Vast.ai, bare-metal, a homelab…), then in a test:

func TestMyProvider(t *testing.T) {
    capacity.RunConformance(t, func(t *testing.T) capacity.Provider { return newMyProvider(t) }, model, instanceType)
}

Provenance

Extracted from Otium, where it's the boundary between the scheduler and the compute it runs on. The interface is open precisely so anyone can contribute capacity; the tuned placement/pricing strategy that decides which offer to take stays in Otium.

License

Apache-2.0.

Documentation

Overview

Package capacity abstracts where Otium's workers run. A Provider quotes live prices, launches and terminates workers, and reports interruptions. AWS Spot is the first real implementation; the interface anticipates GCP/Azure/Oracle preemptible, RunPod, Vast.ai, bare metal, and homelab providers. A fake provider backs tests and local end-to-end runs of the provisioning loop.

The scheduler combines the model catalog (model -> required spec -> candidate instance types) with the pricing index (historical baseline) and a Provider's live Quote to decide whether, where, and how much to provision. Quote — not the index — is what we provision against and pay. See docs/architecture.md.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AuditedInstance 

type AuditedInstance struct {
	Provider     string    // the cloud ("aws", "gcp", …), for cross-cloud labels/routing
	Region       string    // the region/zone it runs in
	InstanceID   string    // the provider machine id
	InstanceType string    // e.g. "g4dn.xlarge"
	Name         string    // best-effort human label (Name tag / label); "" if none
	Owned        bool      // carries Otium's ownership tag/label — i.e. the reaper's domain
	LaunchedAt   time.Time // when it started; zero if the provider didn't report it
}

AuditedInstance is one running compute instance in a provider's account/project — EVERY instance, not just the ones Otium launched. It is what an account-wide cost-leak watchdog scans, so it can flag anything running longer than it should (a leaked image-build VM, a box someone forgot) that the ownership-scoped reaper never sees.

type Fake 

type Fake struct {

	// OnLaunch, if set, is called after a worker is recorded as launched — the seam
	// that lets a local harness start an in-process worker bound to spec.Model.
	OnLaunch func(h WorkerHandle)
	// LaunchErr, if set, is consulted at the start of Launch: a non-nil return fails that launch
	// without recording it — the seam a test uses to simulate InsufficientInstanceCapacity for a
	// specific offer/AZ so the provisioner's cheapest-first offer fallback can be exercised.
	LaunchErr func(offer Offer) error
	// QuoteErr, if non-nil, is returned by Quote without quoting — the seam for exercising a
	// MultiCloudProvider fanning out over a cloud that's unreachable.
	QuoteErr error
	// ListOwnedErr, if non-nil, is returned by ListOwned without listing — the seam for the
	// fail-closed behaviour of aggregated orphan reconciliation.
	ListOwnedErr error
	// OnTerminate, if set, is called when a worker is terminated.
	OnTerminate func(h WorkerHandle)
	// contains filtered or unexported fields
}

Fake is an in-process Provider for tests and local end-to-end runs of the provisioning loop. It quotes a configured set of offers, tracks launched and terminated workers, and — via an optional launch hook — can actually start an in-process worker so the full provision -> drain -> terminate loop runs without cloud. It is safe for concurrent use.

func NewFake 

func NewFake() *Fake

NewFake returns a fake provider. Register offers per model with SetOffers.

func (*Fake) AddOwnedInstance 

func (f *Fake) AddOwnedInstance(oi OwnedInstance)

AddOwnedInstance injects an OwnedInstance that has NO matching live handle — an orphan, for testing reconciliation (an instance the provider knows but the registry doesn't). It is returned by ListOwned and removed by Terminate.

func (*Fake) Launch 

func (f *Fake) Launch(_ context.Context, id string, offer Offer, spec WorkerSpec) (WorkerHandle, error)

Launch implements Provider. It uses the caller-supplied id as the handle id and synthesizes an InstanceID so Terminate (and a registry-reconstructed handle) works.

func (*Fake) LaunchedCount 

func (f *Fake) LaunchedCount() int

LaunchedCount returns the total number of Launch calls (tests).

func (*Fake) ListOwned 

func (f *Fake) ListOwned(_ context.Context) ([]OwnedInstance, error)

ListOwned implements Provider: the fake's currently-live handles as OwnedInstances, so orphan-reconciliation logic is exercisable without real cloud inventory.

func (*Fake) LiveCount 

func (f *Fake) LiveCount() int

LiveCount returns the number of workers currently launched and not terminated.

func (*Fake) Name 

func (f *Fake) Name() string

Name implements Provider. Lock-free by design: SetName is setup-only, and Launch reads the field directly while already holding the lock (calling Name() there would self-deadlock).

func (*Fake) Quote 

func (f *Fake) Quote(_ context.Context, req Requirement) ([]Offer, error)

Quote implements Provider. It returns the offers registered for req.Model (or the fallback), filtered to the requirement's candidate instance types if any are given.

func (*Fake) SetClock 

func (f *Fake) SetClock(now func() time.Time)

SetClock overrides the time source (tests).

func (*Fake) SetName 

func (f *Fake) SetName(name string)

SetName overrides the provider name Name() reports. Tests use it to give sibling fakes distinct names so a MultiCloudProvider can route between them (its routing key is Name).

func (*Fake) SetOffers 

func (f *Fake) SetOffers(model string, offers []Offer)

SetOffers registers the offers Quote returns for a model. An empty model registers the fallback used when a model has no specific offers.

func (*Fake) Terminate 

func (f *Fake) Terminate(_ context.Context, h WorkerHandle) error

Terminate implements Provider.

type Gate 

type Gate interface {
	ProvisionAllowed(ctx context.Context, provider string) (bool, error)
}

Gate reports whether a cloud is currently cleared to launch workers. pricing.ProviderGate implements it over the provider registry.

type GatedProvider 

type GatedProvider struct {
	// contains filtered or unexported fields
}

GatedProvider wraps one cloud's Provider with an operator kill switch, consulted on every Quote and Launch so flipping it takes effect within one provisioner tick rather than on the next restart — which is the whole point of having it during an incident.

Terminate and ListOwned are deliberately NOT gated. Disabling a cloud must never block cleaning up its instances: that would leak billing at exactly the moment an operator is trying to stop it, and the reaper depends on both to find orphans.

Wrap each cloud BEFORE combining them, so a MultiCloudProvider keeps shopping the others: 

multi, _ := NewMultiCloudProvider("clouds",
    NewGatedProvider("aws", awsProvider, gate),
    NewGatedProvider("gcp", gcpProvider, gate))

func (*GatedProvider) Launch 

func (g *GatedProvider) Launch(ctx context.Context, id string, offer Offer, spec WorkerSpec) (WorkerHandle, error)

Launch refuses when the cloud is gated off. Unlike Quote this is an error, not a silent no-op: something already decided to spend money here, and swallowing that would leave the provisioner believing a worker exists.

func (*GatedProvider) ListOwned 

func (g *GatedProvider) ListOwned(ctx context.Context) ([]OwnedInstance, error)

ListOwned is never gated — see the type comment.

func (*GatedProvider) Name 

func (g *GatedProvider) Name() string

Name implements Provider, returning the wrapped provider's name so offers, routing, and logs are unchanged by the wrapper.

func (*GatedProvider) Quote 

func (g *GatedProvider) Quote(ctx context.Context, req Requirement) ([]Offer, error)

Quote returns no offers when the cloud is gated off. No offers — rather than an error — is the honest answer: the cloud genuinely has no capacity available to us right now, and the score treats it exactly as it would a cloud that quoted nothing.

func (*GatedProvider) Terminate 

func (g *GatedProvider) Terminate(ctx context.Context, h WorkerHandle) error

Terminate is never gated — see the type comment.

type InstanceAuditor 

type InstanceAuditor interface {
	Name() string
	AuditInstances(ctx context.Context) ([]AuditedInstance, error)
}

InstanceAuditor lists ALL running instances in a provider's account (across the regions it covers). It is a SEPARATE, OPTIONAL capability from Provider — provider-agnostic on purpose, so the same leak watchdog works for AWS today and GCP/Azure/OCI later with no change to the alert path. It deliberately ignores ownership, so it catches leaks that ListOwned (and thus the reaper) cannot. A provider that doesn't implement it is simply skipped by the auditor.

type Kind 

type Kind string

Kind distinguishes interruptible spot capacity from stable on-demand. 

const (
	KindSpot     Kind = "spot"
	KindOnDemand Kind = "on_demand"
)

type MultiCloudProvider 

type MultiCloudProvider struct {
	// contains filtered or unexported fields
}

MultiCloudProvider presents providers from several clouds (each typically a MultiRegionProvider for one cloud) as a single Provider, so a caller can Quote the combined inventory of every configured cloud at once and pick among the offers itself. Which clouds exist is data, wired in at construction, never compiled in.

Routing is by provider name: Quote fans out and concatenates (each sub-provider's offers already carry their own Provider); Launch routes on the offer's Provider; Terminate on the handle's Provider; ListOwned aggregates and stamps each instance with the provider it was found under so a later terminate can be addressed. Unlike MultiRegionProvider — whose per-region children deliberately SHARE one Name — the sub-providers here must have DISTINCT Names, because the Name is the routing key. Composition is expected: each sub-provider is typically itself a MultiRegionProvider, giving two-level routing (cloud -> region).

func NewMultiCloudProvider 

func NewMultiCloudProvider(name string, subs ...Provider) (*MultiCloudProvider, error)

NewMultiCloudProvider groups per-cloud providers under one aggregate name. subs must be non-empty, every provider non-nil, and their Names distinct and non-empty (the Name is the Launch/Terminate/ListOwned routing key).

func (*MultiCloudProvider) AuditInstances 

func (m *MultiCloudProvider) AuditInstances(ctx context.Context) ([]AuditedInstance, error)

AuditInstances implements InstanceAuditor by fanning the account-wide scan across every cloud. A sub-provider that doesn't implement InstanceAuditor is skipped (that cloud contributes nothing rather than failing the sweep); Provider is stamped so a leak is attributable.

func (*MultiCloudProvider) Launch 

func (m *MultiCloudProvider) Launch(ctx context.Context, id string, offer Offer, spec WorkerSpec) (WorkerHandle, error)

Launch routes to the cloud that produced the offer.

func (*MultiCloudProvider) ListOwned 

func (m *MultiCloudProvider) ListOwned(ctx context.Context) ([]OwnedInstance, error)

ListOwned aggregates owned instances across clouds, stamping each with the provider it was found under so orphan reconciliation can route the terminate (the sub-provider need not know its own cloud name). A per-cloud error fails the whole call — orphan reconciliation must see a complete picture or none, never a partial one that could miss a cost leak. Region is left exactly as the sub-provider set it (e.g. a MultiRegionProvider child).

func (*MultiCloudProvider) Name 

func (m *MultiCloudProvider) Name() string

Name implements Provider.

func (*MultiCloudProvider) Providers 

func (m *MultiCloudProvider) Providers() []string

Providers returns the configured sub-provider names, sorted — for logging and a deterministic fan-out order.

func (*MultiCloudProvider) Quote 

func (m *MultiCloudProvider) Quote(ctx context.Context, req Requirement) ([]Offer, error)

Quote fans the requirement out to every cloud and concatenates the offers, each already stamped with its own Provider. A cloud that errors is skipped so one unreachable cloud can't blind the scheduler to the others; an error surfaces only when EVERY cloud fails.

func (*MultiCloudProvider) Terminate 

func (m *MultiCloudProvider) Terminate(ctx context.Context, h WorkerHandle) error

Terminate routes to the cloud named on the handle.

type MultiRegionProvider 

type MultiRegionProvider struct {
	// contains filtered or unexported fields
}

MultiRegionProvider presents a set of per-region Providers (one client per region) as a single Provider, so a caller can Quote capacity across every configured region and pick among the offers itself. It is region-agnostic: which regions exist is data, wired in at construction, never compiled in.

Routing is by region: Quote fans out and tags nothing (each sub-provider's offers already carry their Region); Launch/Terminate route on the offer/handle Region; ListOwned aggregates and stamps each instance with the region it was found in so a later terminate can be addressed. All sub-providers share one provider Name (e.g. "aws-spot").

func NewMultiRegionProvider 

func NewMultiRegionProvider(name string, byRegion map[string]Provider) (*MultiRegionProvider, error)

NewMultiRegionProvider groups per-region providers under one name. byRegion must be non-empty and every value non-nil.

func (*MultiRegionProvider) AuditInstances 

func (m *MultiRegionProvider) AuditInstances(ctx context.Context) ([]AuditedInstance, error)

AuditInstances implements InstanceAuditor by fanning the account-wide scan across every region's provider. A region provider that doesn't implement InstanceAuditor contributes nothing rather than failing the whole sweep (the auditor is best-effort visibility).

func (*MultiRegionProvider) Launch 

func (m *MultiRegionProvider) Launch(ctx context.Context, id string, offer Offer, spec WorkerSpec) (WorkerHandle, error)

Launch routes to the offer's region.

func (*MultiRegionProvider) ListOwned 

func (m *MultiRegionProvider) ListOwned(ctx context.Context) ([]OwnedInstance, error)

ListOwned aggregates owned instances across regions, stamping each with the region it was found in (its sub-provider need not know its own region) so orphan reconciliation can route the terminate. A per-region error fails the whole call — orphan reconciliation must see a complete picture or none, never a partial one that could miss a cost leak.

func (*MultiRegionProvider) Name 

func (m *MultiRegionProvider) Name() string

Name implements Provider.

func (*MultiRegionProvider) Quote 

func (m *MultiRegionProvider) Quote(ctx context.Context, req Requirement) ([]Offer, error)

Quote fans the requirement out to every region and concatenates the offers. A region that errors is skipped (logged by the caller via the returned error only when ALL regions fail) so one bad region can't blind the scheduler to the others.

func (*MultiRegionProvider) Regions 

func (m *MultiRegionProvider) Regions() []string

Regions returns the configured regions, sorted — for logging and the provisioner's candidate set.

func (*MultiRegionProvider) Terminate 

func (m *MultiRegionProvider) Terminate(ctx context.Context, h WorkerHandle) error

Terminate routes to the handle's offer region.

type Offer 

type Offer struct {
	Provider     string
	InstanceType string
	Region       string
	Zone         string
	Kind         Kind
	PricePerHour float64
	Currency     string
}

Offer is a live, quotable unit of capacity — what the scheduler actually provisions against and the price we actually pay.

type OwnedInstance 

type OwnedInstance struct {
	WorkerID   string // the otium:worker-id tag (the registry/handle id), may be empty on a partial launch
	InstanceID string // the provider machine id, always present — enough to Terminate
	Model      string
	Region     string // the region it runs in — set by MultiRegionProvider so a terminate can be routed
	Provider   string // the cloud it runs on — set by MultiCloudProvider so a cross-cloud terminate can be routed
	LaunchedAt time.Time
}

OwnedInstance is the provider's own view of one Otium-launched machine, discovered from provider inventory (the ownership tag) rather than the durable registry. It is the input to orphan reconciliation: an owned instance whose WorkerID is not live in the fleet registry is a cost leak (a launch the control plane lost track of after a crash or a dropped Launch response) and gets terminated.

type Provider 

type Provider interface {
	// Name identifies the provider, e.g. "aws-spot" or "fake".
	Name() string
	// Quote returns live offers able to satisfy the requirement, cheapest-relevant
	// first is not required — the scheduler scores them.
	Quote(ctx context.Context, req Requirement) ([]Offer, error)
	// Launch provisions a worker onto an offer under the caller-supplied id (used as the
	// instance's ownership tag and idempotency token) and returns a handle once it is
	// accepted (not necessarily ready). The handle carries the provider InstanceID.
	Launch(ctx context.Context, id string, offer Offer, spec WorkerSpec) (WorkerHandle, error)
	// Terminate tears a worker down, addressed by the handle's InstanceID.
	Terminate(ctx context.Context, h WorkerHandle) error
	// ListOwned returns every not-yet-terminated instance the provider knows this Otium
	// owns (by ownership tag). It is the safety net: the provisioner cross-references it
	// against the durable registry and terminates anything the control plane isn't
	// tracking, so no instance can bill forever after a state loss.
	ListOwned(ctx context.Context) ([]OwnedInstance, error)
}

Provider is a source of disposable compute. Implementations must be safe for concurrent use. Liveness is no longer the provider's concern: the durable worker registry (pkg/fleet) and worker heartbeats own it (see docs/worker-registry.md), so a provider holds no per-worker state — Launch returns everything Terminate later needs.

func NewGatedProvider 

func NewGatedProvider(cloud string, inner Provider, gate Gate, log *slog.Logger) Provider

NewGatedProvider wraps inner with the gate for cloud. A nil gate returns inner unwrapped, so callers with no registry wired are unaffected.

type Requirement 

type Requirement struct {
	Model         string
	InstanceTypes []string
	Region        string
}

Requirement describes the capacity a model needs: the candidate instance types (from the catalog) and where to look.

type WorkerHandle 

type WorkerHandle struct {
	// ID is the caller-supplied worker id (passed to Launch). It is the worker's identity
	// across the registry, the per-worker token, and the lease/heartbeat path.
	ID string
	// InstanceID is the provider's own id for the underlying machine (e.g. the EC2 instance
	// id). It lets a worker be terminated or reconciled straight from durable state, with no
	// in-provider tracking. May equal ID for providers with no separate notion.
	InstanceID string
	Provider   string
	Offer      Offer
	Spec       WorkerSpec
	LaunchedAt time.Time
}

WorkerHandle identifies a launched worker so it can be tracked and terminated.

type WorkerSpec 

type WorkerSpec struct {
	// Model is the model the worker should serve.
	Model string
	// Image is the AMI ID or image reference for the instance (e.g. "ami-0abc123"). Zero
	// means the provider uses its own default; the fake ignores this field.
	Image string
	// ImageVariant selects which AMI lineage the provider resolves when Image is empty: the
	// newest self-owned AMI tagged otium:ami=<ImageVariant>. Zero means the default "worker"
	// lineage. It lets one model require a different baked runtime stack (e.g. a nightly vLLM
	// for a brand-new architecture) without moving the stable worker AMI. Ignored when Image
	// is set (an explicit id wins) and by providers that don't resolve images (the fake).
	ImageVariant string
	// UserData is the cloud-init or bootstrap script payload (base64-encoded for AWS).
	// Zero means no user-data is passed; the fake ignores this field.
	UserData string
	// Env is a map of additional environment variables to inject into the worker process.
	// Zero or nil means no extra variables; the fake ignores this field.
	Env map[string]string
	// InstanceProfile is the IAM instance profile name or ARN granting the worker its
	// AWS permissions (e.g. "otium-worker-profile"). Zero means no profile is attached;
	// the fake ignores this field.
	InstanceProfile string
}

WorkerSpec describes the worker to launch onto an Offer.

Source Files

View all Source files

Directories

Path Synopsis
Package aws implements capacity.Provider for AWS EC2 Spot instances.
 Package aws implements capacity.Provider for AWS EC2 Spot instances.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.