shard

package
v0.14.3-dev Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package shard orchestrates a Prima.cpp pipelined-ring shard over the mesh forwarder: serving a model bigger than any single node by splitting its layers across paired mesh peers. Each rank binds its zmq data+signal ports; the mesh forwarder carries the ring (rank→next) and the master-spoke (every rank→rank0) over hole-punched, encrypted peer links, so no rank touches the network directly — the firewall / ssh-proxy / elevation pain of the raw cross-machine proof disappears. cloudbox brokers the introduction; bytes go peer-to-peer.

Prima topology (verified from prima.cpp src/llama.cpp, map_rank_to_port = base+rank):

  • each rank BINDS data(DataPort+rank) and signal(SignalPort+rank)
  • each rank CONNECTS to next-rank data+signal (the ring) and rank0 data (the master-spoke; rank0 dials itself = loopback, skipped)

This replaces the earlier llama.cpp-RPC *star* skeleton — Prima.cpp (proven cross-machine 2026-06-29) is the engine. See docs/distributed-inference-v0-plan.md and docs/distributed-inference-sota-map.md.

Index

Constants

View Source 
const (
	DefaultDataPort   = 9000
	DefaultSignalPort = 10000

	// Mesh-forwarder service names this host's ring channels are exposed under.
	DataService   = "shard-data"
	SignalService = "shard-signal"
)

Prima.cpp port bases (src/llama.cpp). DataPort is the `--data-port` default; SignalPort is fixed in prima (no CLI flag). Rank i uses base+i for each.

View Source 
const ControlService = "shard-ctl"

ControlService is the mesh-forwarder service name the shard-control endpoint is exposed under: a leader reaches a worker's control endpoint over the mesh to tell it to stand up its rank.

Variables

This section is empty.

Functions

func Apply added in v0.11.0 

func Apply(f Forwarder, p *HostPlan) (cleanup func(), err error)

Apply wires this host's plan into the mesh forwarder: Expose every local service, then open every forward listener. Fail-closed — any Listen error unwinds everything already opened, so a half-formed ring never exists. The returned cleanup closes the listeners and unexposes the services.

Types

type Decider added in v0.12.0 

type Decider func(modelBytes uint64, nodes []NodeCapacity) Decision

Decider chooses whether to shard a model and which node leads, from the gathered fleet state. The default is DecideShard — deterministic, most-VRAM leads. It is the *bootstrap*: always available, no dependency, so a shard can always form. The brain (the pooled-LLM Refiner) refines this choice when wired; the deterministic Decider is what it bootstraps from.

type Decision added in v0.12.0 

type Decision struct {
	ShouldShard bool   // true → too big for one node but fits the pooled total
	Leader      string // host that should lead (most capacity); "" when not sharding
	Reason      string
}

Decision is the auto-trigger's verdict for a model of a given size against a set of candidate nodes.

func DecideShard added in v0.12.0 

func DecideShard(modelBytes uint64, nodes []NodeCapacity) Decision

DecideShard decides whether a model of modelBytes should be served sharded across the candidate nodes, and which node leads:

  • fits on the single biggest node → no shard (the pool routes to it)
  • bigger than one node, ≤ pooled total → shard; leader = most-capacity node
  • bigger than the pooled total → can't serve (no shard)

Leader = the most-capacity (most-VRAM) node — it holds the largest contiguous layer span and drives generation — matching the zero-config "most-VRAM host leads" default.

type Expose added in v0.11.0 

type Expose struct {
	Service string `json:"service"`
	Addr    string `json:"addr"` // 127.0.0.1:(base+myRank)
}

Expose is a loopback service this host publishes over the mesh (its bound prima port).

type FormRequest added in v0.12.0 

type FormRequest struct {
	Ring    Ring     `json:"ring"`
	MyRank  int      `json:"my_rank"`
	Model   string   `json:"model"`
	APIPort int      `json:"api_port"`
	Extra   []string `json:"extra,omitempty"`
}

FormRequest is the leader→worker control message: stand up your rank in this ring for this model. Binaries are NOT carried — the worker resolves its own.

type Forward added in v0.11.0 

type Forward struct {
	LocalAddr string `json:"local_addr"` // 127.0.0.1:(port prima connects to)
	PeerID    string `json:"peer_id"`
	Service   string `json:"service"`
}

Forward is a local listener this host opens that bridges to a peer's exposed service — bound at the EXACT loopback port prima will dial.

type Forwarder 

type Forwarder interface {
	Expose(service, loopbackAddr string)
	Unexpose(service string)
	Listen(localAddr, peerID, service string) (net.Listener, error)
}

Forwarder is the subset of *mesh.Forwarder the shard needs. The daemon passes the real forwarder (which satisfies this); the interface keeps the wiring unit-testable without the whole mesh surface.

type HostPlan added in v0.11.0 

type HostPlan struct {
	MyRank   int       `json:"my_rank"`
	World    int       `json:"world"`
	DataPort int       `json:"data_port"`
	Exposes  []Expose  `json:"exposes"`
	Forwards []Forward `json:"forwards"`
	// PrimaArgs is the distributed-inference flag set. The launcher appends the
	// model (`-m`), and for rank 0 the prompt (`-p`/`-n`); workers add none.
	PrimaArgs []string `json:"prima_args"`
}

HostPlan is the computed mesh wiring + prima distributed args for one host.

func (*HostPlan) FullArgs added in v0.11.0 

func (p *HostPlan) FullArgs(modelPath string, extra ...string) []string

FullArgs builds the complete prima argv: the distributed flags, the model, any caller extras (e.g. --prefetch, --gpu-mem, and for rank 0 the -p/-n prompt).

func (*HostPlan) LaunchConfigFor added in v0.12.0 

func (p *HostPlan) LaunchConfigFor(sc ServeConfig) LaunchConfig

LaunchConfigFor builds this host's LaunchConfig from its plan + a ServeConfig. rank 0 → the server binary with --host/--port (the served OpenAI endpoint); any other rank → the worker binary for its layer span. The distributed ring flags come from the plan (PrimaArgs); this only chooses the binary + the role-specific serve flags.

type LaunchConfig added in v0.11.0 

type LaunchConfig struct {
	BinaryPath string    // path to the prima llama-cli / llama-server binary
	ModelPath  string    // -m <model>
	Extra      []string  // --prefetch, --gpu-mem, and for rank 0 the -p/-n prompt
	LogWriter  io.Writer // prima stdout+stderr sink (nil → discarded)
}

LaunchConfig configures a prima process launch on this host.

type LocalModel added in v0.12.0 

type LocalModel struct {
	Name  string
	Bytes uint64
}

LocalModel is a model present on this node, with its on-disk size.

type Manager added in v0.12.0 

type Manager struct {
	// contains filtered or unexported fields
}

Manager keeps a current candidate shard Ring up to date: it periodically discovers the reachable same-LAN owner peers and assembles a launch-ready ring. It does NOT form a shard by itself — standing the ring up is gated on a too-big model (the auto-trigger, v1d); the manager just keeps the ring ready.

func NewManager added in v0.12.0 

func NewManager(cfg ManagerConfig) *Manager

NewManager builds a shard manager. Defaults: 30s discover interval, the default slog logger.

func (*Manager) ActiveModel added in v0.12.0 

func (m *Manager) ActiveModel() string

ActiveModel returns the model this node is currently serving via a shard, or "" if none — the name the pool advertises so cloudbox routes requests for it to this (leader) node.

func (*Manager) Form added in v0.12.0 

func (m *Manager) Form(ctx context.Context, ring *Ring, myRank int, sc ServeConfig) error

Form launches THIS node's part of a shard for the given ring + serve config (the caller — the trigger — decides ring/rank/model/when). The leader serves the model's OpenAI endpoint; workers serve their layer span. Recording the served model lets the pool advertise it (ActiveModel). Forming again replaces the previous shard.

func (*Manager) LastExit added in v0.12.12 

func (m *Manager) LastExit() string

LastExit returns a description of the most recent prima exit on this node (model + error), or "" if none. It's surfaced in the status report so a worker-rank crash is visible over the mesh — no ssh into the box needed.

func (*Manager) LocalStatus added in v0.12.0 

func (m *Manager) LocalStatus() StatusReport

LocalStatus builds this node's report from live manager state (no network).

func (*Manager) MaybeShard added in v0.12.0 

func (m *Manager) MaybeShard(ctx context.Context, localModels []LocalModel, localBytes uint64, apiPort int) error

MaybeShard is the auto-trigger: for the first local model too big to serve on this node alone (but a same-LAN ring exists to spread it), it orchestrates a shard with this node as leader. Idempotent — skips the already-active model. The daemon calls this with the local ollama models + this node's memory budget.

func (*Manager) Orchestrate added in v0.12.0 

func (m *Manager) Orchestrate(ctx context.Context, model string, apiPort int, extra []string) error

Orchestrate forms a shard for the model across the current ring with THIS node as leader (rank 0): it tells every worker (over the mesh shard-control) to stand up its rank, then forms its own. The caller (the trigger) decides when and which model. Fail-fast: a worker that won't form aborts the whole form.

func (*Manager) PeerLog added in v0.12.16 

func (m *Manager) PeerLog(ctx context.Context, peer ShardPeer) (string, error)

PeerLog forwards to a peer's shard-control /log over the mesh and returns the tail of its captured prima-rank logs — the worker's own exit reason surfaced on the leader, no ssh. Mirrors PingPeer, with a short HTTP timeout.

func (*Manager) PingPeer added in v0.12.0 

func (m *Manager) PingPeer(ctx context.Context, peer ShardPeer) (*StatusReport, error)

PingPeer forwards to a peer's shard-control /status over the mesh and returns its report — app-level reachability + readiness, no ssh. The caller resolves host→PeerID (cloudbox peer/connect), so it works for any paired peer, not just same-LAN ones.

func (*Manager) RecentPrimaLogs added in v0.12.16 

func (m *Manager) RecentPrimaLogs(maxLines int) string

RecentPrimaLogs returns the tail of every captured prima-rank log on this node: each <logDir>/prima-rank<N>.log prefixed with a "==== <name> ====" header and its last maxLines lines. Returns "" when no log dir is configured or none exist. This is the self-diagnosis surface — the exit reason a crashed rank left behind, readable over the mesh (via /log) with no ssh into the box.

func (*Manager) Ring added in v0.12.0 

func (m *Manager) Ring() *Ring

Ring returns a snapshot of the current candidate ring (nil if there are no same-LAN peers to shard with).

func (*Manager) Run added in v0.12.0 

func (m *Manager) Run(ctx context.Context) error

Run refreshes the candidate ring immediately, then on every interval, until ctx is cancelled.

func (*Manager) ServeControl added in v0.12.0 

func (m *Manager) ServeControl() (func(), error)

ServeControl runs the shard-control HTTP handler on a fresh loopback listener and exposes it over the mesh as ControlService, so a leader can drive this node to form its rank. The returned cleanup unexposes + shuts it down.

func (*Manager) Stop added in v0.12.0 

func (m *Manager) Stop()

Stop tears down the active shard on this node (if any).

func (*Manager) TellLead added in v0.12.2 

func (m *Manager) TellLead(ctx context.Context, peer ShardPeer, model string, apiPort int) error

TellLead tells a peer (over the mesh) to LEAD a shard for the model: that node becomes the leader, self-provisions, and orchestrates its workers. This is the fleet trigger — an agent (or, later, cloudbox on a pool request) starts a shard on any node with no ssh, and the system self-drives from there.

type ManagerConfig added in v0.12.0 

type ManagerConfig struct {
	Self      ShardPeer      // this host (label + its own libp2p peer id)
	Forwarder Forwarder      // the mesh forwarder (the data plane)
	Peers     PeerDiscoverer // same-LAN owner-peer source
	Interval  time.Duration  // discover cadence (0 → 30s)
	Logger    *slog.Logger
	Bins      ServeBins // this node's prima binaries (server + worker)
	// LocalLoad yields this node's local models (with sizes) + its model-memory
	// budget; when set, the discover loop auto-triggers a shard for a too-big
	// model (MaybeShard). nil → no auto-trigger.
	LocalLoad func() ([]LocalModel, uint64)
	APIPort   int // OpenAI port for a leader-served shard (0 → 11434)
	// Provision ensures the model (+ engine binaries) are present locally,
	// fetching them with no human staging, and returns the GGUF path prima loads.
	// nil → identity (model name used as-is; for tests + already-staged hosts).
	Provision func(ctx context.Context, modelName string) (string, error)
	// Refiner, when set, lets the pooled LLM (the brain) refine the leader
	// election. nil → the deterministic bootstrap (most-VRAM) stands.
	Refiner brain.Refiner
	// LogDir, when set, is where each rank's prima stdout+stderr is captured
	// (<LogDir>/prima-rank<N>.log) — the exit reason when a shard process dies.
	LogDir string
}

ManagerConfig configures the shard manager.

type Member added in v0.11.0 

type Member struct {
	Rank   int    `json:"rank"`
	Host   string `json:"host"`    // label for logging
	PeerID string `json:"peer_id"` // libp2p peer id; "" = this host
}

Member is one rank in the pipelined ring. Rank 0 is the leader (master + prompt driver). PeerID is the member's libp2p peer id; empty marks THIS host.

type NodeCapacity added in v0.12.0 

type NodeCapacity struct {
	Host  string
	Bytes uint64
}

NodeCapacity is one candidate node's usable model-memory budget (VRAM plus any spillover RAM the engine can use) for shard placement.

type PeerDiscoverer added in v0.12.0 

type PeerDiscoverer interface {
	SameLANPeers(ctx context.Context) ([]ShardPeer, error)
}

PeerDiscoverer yields the reachable same-LAN owner peers eligible as shard workers. The real implementation wraps the peer-plane (same-LAN/tier filter) + cloudbox peer/connect (peer-id resolution); tests inject a fake.

type Ring added in v0.11.0 

type Ring struct {
	Members    []Member `json:"members"`
	DataPort   int      `json:"data_port,omitempty"`   // 0 → DefaultDataPort
	SignalPort int      `json:"signal_port,omitempty"` // 0 → DefaultSignalPort
}

Ring is a full shard placement: one Member per rank (any order) plus the port bases. Zero ports default to the prima defaults.

func (Ring) PlanFor added in v0.11.0 

func (r Ring) PlanFor(myRank int) (*HostPlan, error)

PlanFor computes the mesh wiring + prima args for the host running rank myRank. Pure — no I/O — so the ring logic is fully unit-testable.

type ServeBins added in v0.12.0 

type ServeBins struct {
	ServerBin string // prima llama-server (leader, rank 0)
	WorkerBin string // prima llama-cli (workers)
}

ServeBins are this node's Prima binaries. Resolved per host — paths differ by OS/install, so a worker always uses its OWN, never the leader's.

type ServeConfig added in v0.12.0 

type ServeConfig struct {
	Model     string   // -m <model gguf>
	ServerBin string   // prima llama-server path (leader, rank 0)
	WorkerBin string   // prima llama-cli path (workers)
	APIHost   string   // leader OpenAI bind host (default 127.0.0.1)
	APIPort   int      // leader OpenAI port
	Extra     []string // engine extras: --prefetch, --gpu-mem, …
}

ServeConfig describes how a node launches its shard process to SERVE a model: the leader (rank 0) runs prima's llama-server (the OpenAI endpoint clients hit); every other rank runs llama-cli over its layer span. The OpenAI port is loopback-only — cloudbox reaches it via the existing ollama-style proxy, and the pool routes to it once advertised.

type Session added in v0.11.0 

type Session struct {
	// contains filtered or unexported fields
}

Session is a running shard participant on this host: the prima process plus the mesh wiring it rides, torn down together by Stop.

func Start added in v0.11.0 

func Start(ctx context.Context, f Forwarder, plan *HostPlan, cfg LaunchConfig) (*Session, error)

Start wires this host's plan into the mesh forwarder, then launches prima against the local loopback ports that wiring owns (prima's --master/--next are 127.0.0.1, so every connection it makes is a mesh-forwarded loopback). It is fail-closed: if prima won't start, the mesh wiring is unwound before returning, so a half-formed shard never lingers.

func (*Session) Running added in v0.11.0 

func (s *Session) Running() bool

Running reports whether the prima process is still alive.

func (*Session) Stop added in v0.11.0 

func (s *Session) Stop()

Stop kills the prima process, waits for it to reap (draining its log), then tears down the mesh wiring. Idempotent — safe to call more than once and alongside a Wait.

func (*Session) Wait added in v0.11.0 

func (s *Session) Wait() error

Wait blocks until the prima process exits on its own and returns its exit error (nil on a clean exit).

type ShardPeer added in v0.12.0 

type ShardPeer struct {
	Host   string
	PeerID string
}

ShardPeer identifies a shard-ring participant on the mesh: a hostname label plus its libp2p peer id (what the forwarder dials).

type StatusReport added in v0.12.0 

type StatusReport struct {
	Host        string       `json:"host"`
	Models      []LocalModel `json:"models,omitempty"`
	BudgetBytes uint64       `json:"budget_bytes"`
	ServerBin   bool         `json:"server_bin"` // prima llama-server present on disk
	WorkerBin   bool         `json:"worker_bin"` // prima llama-cli present on disk
	ActiveModel string       `json:"active_model,omitempty"`
	RingMembers int          `json:"ring_members"`
	LastExit    string       `json:"last_exit,omitempty"` // most recent prima exit (model + error)
}

StatusReport is a node's shard-readiness, returned over the mesh shard-control /status endpoint. It is the app-level ping/pong: a successful fetch IS the reachability proof (the peer's daemon answered over the mesh), and the body carries the facts the orchestrator — or an operator — needs to decide whether the peer can take a rank, with no ssh into the box.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.