inferkit

Documentation

Overview

Package inferkit provides a provider-agnostic HTTP client for OpenAI-compatible LLM inference APIs. It supports chat completions, streaming (SSE), and embeddings. Works with OpenAI, DeepInfra, Groq, Ollama (OpenAI-compat mode), or any server implementing /v1/chat/completions and /v1/embeddings.

Index

• Constants
• type ChatChunk
• type ChatRequest
• type ChatResponse
• type Client
  • func New(cfg Config, opts ...Option) *Client
  • func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)
  • func (c *Client) ChatStream(ctx context.Context, req ChatRequest) (*ResponseMeta, <-chan ChatChunk, error)
  • func (c *Client) Embed(ctx context.Context, req EmbedRequest) (*EmbedResponse, error)
  • func (c *Client) Ping(ctx context.Context) error
• type Config
• type EmbedRequest
• type EmbedResponse
• type Message
• type Option
  • func WithCircuitBreaker(name string, threshold int, cooldown time.Duration) Option
  • func WithHTTPClient(hc *http.Client) Option
  • func WithRetry(maxAttempts int, baseDelay time.Duration) Option
• type ResponseFormat
• type ResponseMeta
• type Usage

Constants

const (
	OpenAI    = "https://api.openai.com/v1"
	DeepInfra = "https://api.deepinfra.com/v1/openai"
	Groq      = "https://api.groq.com/openai/v1"
)

Provider base URLs for common OpenAI-compatible services.

Types

type ChatChunk

type ChatChunk struct {
	Content      string
	FinishReason string
	Done         bool
	Err          error
}

ChatChunk is a single piece of a streamed chat response.

type ChatRequest

type ChatRequest struct {
	Model          string          // Override per-request (optional, falls back to Config.Model)
	Messages       []Message       //
	Temperature    *float64        //
	MaxTokens      *int            //
	ResponseFormat *ResponseFormat // Structured output: {"type": "json_object"}
	ExtraBody      map[string]any  // Provider-specific top-level fields (typed fields win on conflict)
}

ChatRequest is the payload for a chat completion.

type ChatResponse

type ChatResponse struct {
	Content      string
	FinishReason string
	Usage        Usage
	Meta         ResponseMeta
}

ChatResponse is the result of a chat completion.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is an HTTP client for OpenAI-compatible inference APIs. It uses call.Client for timeout, circuit breaker, Bearer token injection, and OTel tracing on Chat, Embed, and Ping calls. ChatStream uses a plain http.Client to avoid imposing a timeout on long-running streams.

func New

func New(cfg Config, opts ...Option) *Client

New creates a new inference client.

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)

Chat sends a chat completion request and returns the full response.

func (*Client) ChatStream

func (c *Client) ChatStream(ctx context.Context, req ChatRequest) (*ResponseMeta, <-chan ChatChunk, error)

ChatStream sends a streaming chat completion request and returns a channel of chunks. The channel is closed when the stream ends. A final chunk with Done=true is sent before the channel closes on a clean finish.

ChatStream uses a plain http.Client (not call.Client) to avoid imposing a timeout on long-running streams. Auth is set from the configured API key.

func (*Client) Embed

func (c *Client) Embed(ctx context.Context, req EmbedRequest) (*EmbedResponse, error)

Embed generates embeddings for the input texts.

func (*Client) Ping

func (c *Client) Ping(ctx context.Context) error

Ping checks connectivity by hitting the models endpoint.

type Config

type Config struct {
	BaseURL string        `env:"INFER_BASE_URL" default:"https://api.openai.com/v1"`
	APIKey  string        `env:"INFER_API_KEY" required:"false"`
	Model   string        `env:"INFER_MODEL"   required:"false"`
	Timeout time.Duration `env:"INFER_TIMEOUT" default:"120s"`
}

Config holds the client configuration.

type EmbedRequest

type EmbedRequest struct {
	Model     string         // Override per-request
	Input     []string       // Texts to embed
	ExtraBody map[string]any // Provider-specific top-level fields (typed fields win on conflict)
}

EmbedRequest is the payload for an embeddings call.

type EmbedResponse

type EmbedResponse struct {
	Vectors [][]float32 // One vector per input text
	Usage   Usage
	Meta    ResponseMeta
}

EmbedResponse contains the embedding vectors.

type Message

type Message struct {
	Role    string `json:"role"`    // "system", "user", "assistant"
	Content string `json:"content"` //
}

Message is a single message in a chat conversation.

type Option

type Option func(*options)

Option configures a Client.

func WithCircuitBreaker

func WithCircuitBreaker(name string, threshold int, cooldown time.Duration) Option

WithCircuitBreaker enables a circuit breaker that opens after consecutive failures. Delegates to call.CircuitBreaker which provides singleton breakers by name, a proper probing state, and OTel instrumentation.

func WithHTTPClient

func WithHTTPClient(hc *http.Client) Option

WithHTTPClient replaces the underlying *http.Client used by the inference client. This is useful when you need a custom Transport (e.g. SSRF-safe dialer, proxy routing). The custom client is used for both the call.Client (Chat, Embed, Ping) and the streaming HTTP client (ChatStream).

func WithRetry

func WithRetry(maxAttempts int, baseDelay time.Duration) Option

WithRetry enables retry with exponential backoff and jitter for transient errors (429 rate-limit and 5xx). Retry is handled at the inferkit level so that 429 responses are retried (call.Retrier only retries 5xx).

type ResponseFormat

type ResponseFormat struct {
	Type string `json:"type"` // "json_object" or "text"
}

ResponseFormat controls the output format.

type ResponseMeta

type ResponseMeta struct {
	StatusCode int
	Header     http.Header
}

ResponseMeta captures HTTP-level metadata from the inference API response.

type Usage

type Usage struct {
	PromptTokens     int
	CompletionTokens int
	TotalTokens      int
}

Usage reports token consumption.