backend

package
v0.4.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 13, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckInsecureAPIKey 

func CheckInsecureAPIKey(rawURL, apiKey string) error

CheckInsecureAPIKey returns an error if an API key is being sent over plain HTTP to a non-localhost host.

func IsConnectionError added in v0.4.1 

func IsConnectionError(err error) bool

IsConnectionError returns true if the error indicates the model server is unreachable (connection refused, timeout, no route, etc.).

Types

type Backend 

type Backend interface {
	// Name returns the backend type name
	Name() string

	// URL returns the backend endpoint URL
	URL() string

	// Complete sends a prompt and returns the full completion
	Complete(ctx context.Context, req *Request) (*Response, error)

	// Stream sends a prompt and streams tokens via the callback
	Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

	// Health checks if the backend is available
	Health(ctx context.Context) error

	// ListModels returns the models available on the backend.
	// Returns nil, nil if the backend does not support listing.
	ListModels(ctx context.Context) ([]string, error)
}

Backend defines the interface for LLM inference backends

func New 

func New(cfg Config) (Backend, error)

New creates a backend based on the config type

type Config 

type Config struct {
	Type   string // "ollama", "llamacpp", "vllm", "lmstudio", "custom"
	URL    string
	Model  string
	APIKey string // for custom backends that need auth
}

Config holds backend configuration

type Custom 

type Custom struct {
	// contains filtered or unexported fields
}

Custom implements the Backend interface for any HTTP endpoint that accepts a simple JSON request and returns a JSON response.

Expected request format: 

{
  "prompt": "...",
  "max_tokens": 512,
  "temperature": 0.7
}

Expected response format: 

{
  "text": "...",
  "prompt_tokens": 10,
  "completion_tokens": 100
}

func NewCustom 

func NewCustom(cfg Config) (*Custom, error)

NewCustom creates a new custom backend

func (*Custom) Complete 

func (c *Custom) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a prompt and returns the full completion

func (*Custom) Health 

func (c *Custom) Health(ctx context.Context) error

Health checks if the custom backend is available

func (*Custom) ListModels added in v0.4.5 

func (c *Custom) ListModels(ctx context.Context) ([]string, error)

ListModels is not supported for custom backends.

func (*Custom) Name 

func (c *Custom) Name() string

Name returns the backend type

func (*Custom) Stream 

func (c *Custom) Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

Stream is not supported for custom backends; falls back to Complete

func (*Custom) URL added in v0.4.1 

func (c *Custom) URL() string

URL returns the backend endpoint URL

type LMStudio added in v0.4.12 

type LMStudio struct {
	// contains filtered or unexported fields
}

LMStudio implements the Backend interface for LM Studio (OpenAI-compatible API)

func NewLMStudio added in v0.4.12 

func NewLMStudio(cfg Config) (*LMStudio, error)

NewLMStudio creates a new LM Studio backend

func (*LMStudio) Complete added in v0.4.12 

func (l *LMStudio) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a prompt and returns the full completion

func (*LMStudio) Health added in v0.4.12 

func (l *LMStudio) Health(ctx context.Context) error

Health checks if LM Studio is available

func (*LMStudio) ListModels added in v0.4.12 

func (l *LMStudio) ListModels(ctx context.Context) ([]string, error)

ListModels returns all models available in LM Studio.

func (*LMStudio) Name added in v0.4.12 

func (l *LMStudio) Name() string

Name returns the backend type

func (*LMStudio) Stream added in v0.4.12 

func (l *LMStudio) Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

Stream sends a prompt and streams tokens via the callback

func (*LMStudio) URL added in v0.4.12 

func (l *LMStudio) URL() string

URL returns the backend endpoint URL

type LlamaCpp 

type LlamaCpp struct {
	// contains filtered or unexported fields
}

LlamaCpp implements the Backend interface for llama.cpp server

func NewLlamaCpp 

func NewLlamaCpp(cfg Config) (*LlamaCpp, error)

NewLlamaCpp creates a new llama.cpp backend

func (*LlamaCpp) Complete 

func (l *LlamaCpp) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a prompt and returns the full completion

func (*LlamaCpp) Health 

func (l *LlamaCpp) Health(ctx context.Context) error

Health checks if llama.cpp server is available

func (*LlamaCpp) ListModels added in v0.4.5 

func (l *LlamaCpp) ListModels(ctx context.Context) ([]string, error)

ListModels is not supported for llama.cpp (single-model server).

func (*LlamaCpp) Name 

func (l *LlamaCpp) Name() string

Name returns the backend type

func (*LlamaCpp) Stream 

func (l *LlamaCpp) Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

Stream sends a prompt and streams tokens via the callback

func (*LlamaCpp) URL added in v0.4.1 

func (l *LlamaCpp) URL() string

URL returns the backend endpoint URL

type Ollama 

type Ollama struct {
	// contains filtered or unexported fields
}

Ollama implements the Backend interface for Ollama

func NewOllama 

func NewOllama(cfg Config) (*Ollama, error)

NewOllama creates a new Ollama backend

func (*Ollama) Complete 

func (o *Ollama) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a prompt and returns the full completion

func (*Ollama) Health 

func (o *Ollama) Health(ctx context.Context) error

Health checks if Ollama is available and the configured model exists

func (*Ollama) ListModels added in v0.4.5 

func (o *Ollama) ListModels(ctx context.Context) ([]string, error)

ListModels returns all models available in Ollama.

func (*Ollama) Name 

func (o *Ollama) Name() string

Name returns the backend type

func (*Ollama) Stream 

func (o *Ollama) Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

Stream sends a prompt and streams tokens via the callback

func (*Ollama) URL added in v0.4.1 

func (o *Ollama) URL() string

URL returns the backend endpoint URL

type Request 

type Request struct {
	Prompt      string
	MaxTokens   int
	Temperature float64
	TopP        float64
}

Request represents an inference request to a backend

type Response 

type Response struct {
	Text             string
	PromptTokens     int
	CompletionTokens int
}

Response represents an inference response from a backend

type VLLM 

type VLLM struct {
	// contains filtered or unexported fields
}

VLLM implements the Backend interface for vLLM (OpenAI-compatible API)

func NewVLLM 

func NewVLLM(cfg Config) (*VLLM, error)

NewVLLM creates a new vLLM backend

func (*VLLM) Complete 

func (v *VLLM) Complete(ctx context.Context, req *Request) (*Response, error)

Complete sends a prompt and returns the full completion

func (*VLLM) Health 

func (v *VLLM) Health(ctx context.Context) error

Health checks if vLLM is available

func (*VLLM) ListModels added in v0.4.5 

func (v *VLLM) ListModels(ctx context.Context) ([]string, error)

ListModels returns all models available in vLLM.

func (*VLLM) Name 

func (v *VLLM) Name() string

Name returns the backend type

func (*VLLM) Stream 

func (v *VLLM) Stream(ctx context.Context, req *Request, callback func(token string, done bool) error) (*Response, error)

Stream sends a prompt and streams tokens via the callback

func (*VLLM) URL added in v0.4.1 

func (v *VLLM) URL() string

URL returns the backend endpoint URL

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.