inference

Documentation

Overview

Package inference holds the OpenAI-compatible clients: embeddings now, reranking in a later phase.

Index

• type Chat
  • func (c *Chat) Complete(ctx context.Context, system, user string) (string, error)
  • func (c *Chat) CompleteWithImage(ctx context.Context, system, user string, image []byte, mime string) (string, error)
• type Embedder
  • func (e *Embedder) Embed(ctx context.Context, texts []string) ([][]float32, error)
  • func (e *Embedder) WaitReady(ctx context.Context, timeout time.Duration) error
• type Ranked
• type Reranker
  • func (r *Reranker) Rerank(ctx context.Context, query string, documents []string, topN int) ([]Ranked, error)

Types

type Chat

type Chat struct {
	BaseURL    string
	APIKey     string
	Model      string
	MaxTokens  int
	MaxRetries int
	// ExtraBody is merged into every request body (provider-specific knobs
	// like chat_template_kwargs).
	ExtraBody map[string]any
	Client    *http.Client
	Logger    *slog.Logger
}

Chat is a minimal OpenAI-compatible /v1/chat/completions client for index-time work (contextual retrieval). Same failure contract as the embedder: transient errors retry with backoff, everything else propagates.

func (*Chat) Complete

func (c *Chat) Complete(ctx context.Context, system, user string) (string, error)

Complete sends one system+user exchange and returns the trimmed response.

func (*Chat) CompleteWithImage

func (c *Chat) CompleteWithImage(ctx context.Context, system, user string, image []byte, mime string) (string, error)

CompleteWithImage sends a user turn carrying an image (OpenAI content-array format with a base64 data URI). The chat model must be vision-capable.

type Embedder

type Embedder struct {
	BaseURL    string // e.g. http://127.0.0.1:8080/v1
	APIKey     string
	Model      string
	Dimensions int
	BatchSize  int
	MaxRetries int
	Client     *http.Client
	Logger     *slog.Logger
}

Embedder calls an OpenAI-compatible /v1/embeddings endpoint.

Contract, learned the hard way: there is NO fallback vector code path. Every error propagates; callers abort the affected document and the previous index state survives. A dummy-vector fallback once tricked a dimension check into wiping an entire collection.

func (*Embedder) Embed

func (e *Embedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed returns one vector per input text, batching requests internally.

func (*Embedder) WaitReady

func (e *Embedder) WaitReady(ctx context.Context, timeout time.Duration) error

WaitReady polls the endpoint with a real 1-text embedding request until it answers correctly or the timeout expires. Inference servers can take minutes to load models after a reboot; indexing must not start before the endpoint is truly ready.

type Ranked

type Ranked struct {
	Index int
	Score float64
}

Ranked is one rerank result. Score is normalized to (0,1) via sigmoid so raw cross-encoder logits stay comparable with min_score filters.

type Reranker

type Reranker struct {
	BaseURL string
	APIKey  string
	Model   string
	Client  *http.Client
	Logger  *slog.Logger
}

Reranker calls an OpenAI-compatible /v1/rerank endpoint (llama-server, Jina, Cohere-compatible shapes). Reranking is an enhancement: callers must degrade gracefully to fusion order when it fails.

func (*Reranker) Rerank

func (r *Reranker) Rerank(ctx context.Context, query string, documents []string, topN int) ([]Ranked, error)

Rerank scores documents against the query and returns them best-first.