inference

package module
v0.7.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 5, 2026 License: MIT Imports: 16 Imported by: 0

README

LLM Inference for Go

Go Report Card lint test

A single interface in Go to get inference from multiple LLM / AI providers using their official SDKs.

Features at a glance

  • Single normalized interface (ProviderSetAPI) for multiple providers. Current support:

  • Normalized data model in spec/:

    • messages (user / assistant / system/developer instructions are provided via ModelParam.SystemPrompt),
    • text, images, and files, (no audio/video content types yet),
    • tools (function, custom, built-in tools like web search),
    • reasoning / thinking content,
    • streaming events (text + thinking),
    • usage accounting.

  • Streaming support:

    • Text streaming for all providers that support it.
    • Reasoning / thinking streaming where the provider exposes it (Anthropic, OpenAI Responses).

  • Client and Server Tools:

    • Client tools are supported via Function Calling.
    • Anthropic server-side web search.
    • OpenAI Responses web search tool.
    • OpenAI Chat Completions web search via web_search_options.

  • HTTP-level debugging:

    • Pluggable CompletionDebugger interface.
    • A built-in ready to use implementation at: debugclient.HTTPCompletionDebugger:
      • wraps SDK HTTP clients,
      • captures request/response metadata,
      • redacts secrets and sensitive content,
      • attaches a scrubbed debug blob to FetchCompletionResponse.DebugDetails.

Installation

# Go 1.25+
go get github.com/flexigpt/inference-go

Quickstart

Basic pattern:

  1. Create a ProviderSetAPI.
  2. Add one or more providers. Set their API keys.
  3. Send a FetchCompletionRequest.

Examples

Supported providers

Anthropic Messages API

Feature support

Area Supported? Notes
Text input/output yes User and assistant messages mapped to text blocks.
Streaming text yes
Reasoning / thinking yes Thinking/Redacted is supported; redacted is not streamed to caller. Thinking enabled == temperature omitted.
Streaming thinking yes
Images (input) yes Inline base64 (imageData) or remote URLs (imageURL) mapped to Anthropic image blocks.
Files / documents (input) yes PDFs only, via base64 or URL. Plain-text base64 and other MIME types are currently ignored.
Audio/Video input/output no
Tools (function/custom) yes JSON Schema based.
Web search yes Server web search tool use + web search tool-result blocks.
Citations partial URL citations only. Other stateful citations are not mapped.
Metadata / service tiers opaque Not exposed in normalized types; available in debug payload.
Stateful flows no Library focuses on stateless calls only.
Usage data yes Input/Output/Cached. Anthropic doesn't expose Reasoning tokens usage.
  • Behavior for conversational + interleaved reasoning message input
    • Input: No reasoning content in the incoming messages.
      • Action: Build the message list unchanged. If the last user message is a tool_result, force thinking disabled; otherwise, honor the requested thinking setting.
    • Input: All reasoning messages are signed.
      • Action: Build the message list unchanged. If the last user message is a tool_result and the previous assistant message begins with thinking content, force thinking enabled; otherwise, honor the requested thinking setting.
    • Input: Mix of reasoning messages where some include a valid signature thinking and others do not.
      • Action: Retain only the reasoning messages with a valid signature; drop the rest. Apply the above behaviors after this cleanup.
OpenAI Responses API

Feature support

Area Supported? Notes
Text input/output yes Input/output messages fully supported.
Streaming text yes
Reasoning / thinking yes Reasoning outputs are mapped. Reasoning inputs are accepted only as encrypted_content; others are dropped.
Streaming thinking yes
Images (input) yes imageData (base64) or imageURL, with detail low/high/auto, mapped to Responses input_image items.
Files / documents (input) yes fileData (base64) or fileURL mapped to Responses input_file items; works for PDFs and other file MIME types.
Audio/Video input/output no
Tools (function/custom) yes JSON Schema based. Note: custom tool definitions are currently emitted as function tools.
Web search yes Calls are mapped when emitted; results typically surface as citations/annotations in text.
Citations yes URL citations mapped to spec.CitationKindURL.
Metadata / service tiers opaque Not exposed in normalized types; available in debug payload.
Stateful flows no Store is explicitly disabled (Store: false).
Usage data yes Input/Output/Cached/Reasoning.
  • Behavior for conversational + interleaved reasoning message input
    • Input: No reasoning messages.
      • Action: Build the message list unchanged. Honor the requested thinking setting.
    • Input: All reasoning messages are encrypted_content.
      • Action: Build the message list unchanged. Honor the requested thinking setting.
    • Input: Mixed reasoning messages: some are signature-based and some are encrypted_content.
      • Action: Keep only the encrypted_content reasoning; drop the signature-based reasoning.
OpenAI Chat Completions API

Feature support

Area Supported? Notes
Text input/output yes Only the first choice from output is surfaced up.
Streaming text yes
Reasoning / thinking yes Reasoning effort config only; no separate reasoning messages in API.
Streaming thinking no Not exposed by Chat Completions.
Images (input) yes imageData (base64) and imageURL are both supported; base64 is sent as a data URL with detail low/high/auto.
Files / documents (input) yes fileData (base64) only, sent as a data URL; fileURL and stateful file IDs are not used by this adapter.
Audio/Video input/output no
Tools (function/custom) yes JSON Schema based. Note: custom tool definitions are currently emitted as function tools.
Web search yes API doesn't expose a tool; mapped via top-level web_search_options derived from a webSearch ToolChoice.
Citations yes URL citations mapped from annotations.
Metadata / service tiers opaque Not exposed in normalized types; available in debug payload.
Stateful flows no Library focuses on stateless calls only.
Usage data yes Input/Output/Cached/Reasoning.
  • Behavior for conversational + interleaved reasoning message input
    • Reasoning effort config is kept as is.
    • All reasoning input/output messages are dropped as the api doesn't support it.

HTTP debugging

The library exposes a pluggable CompletionDebugger interface:

type CompletionDebugger interface {
    HTTPClient(base *http.Client) *http.Client
    StartSpan(ctx context.Context, info *spec.CompletionSpanStart) (context.Context, spec.CompletionSpan)
}

  • package debugclient includes an implementation that can be readily used as HTTPCompletionDebugger:

    • wraps the provider SDK’s *http.Client,
    • captures and scrubs:
      • URL, method, headers (with secret redaction),
      • query params,
      • request/response bodies (optional, scrubbed of LLM text and large base64),
      • curl command for reproduction,
    • attaches a structured HTTPDebugState to FetchCompletionResponse.DebugDetails.
    • You can then inspect resp.DebugDetails for a given call, or just rely on slog output.

  • Use it via WithDebugClientBuilder:

ps, _ := inference.NewProviderSetAPI(
    inference.WithDebugClientBuilder(func(p spec.ProviderParam) spec.CompletionDebugger {
        return debugclient.NewHTTPCompletionDebugger(&debugclient.DebugConfig{
            LogToSlog: false,
        })
    }),
)

Notes

  • Stateless focus. The design focuses on stateless request/response interactions:

    • no conversation IDs,
    • no file IDs,

  • Opaque / provider‑specific fields.

    • Many provider‑specific fields (error details, service tiers, cache metadata, full raw responses) are only available through the debug payload, not in the normalized spec types.
    • Few of the common needed params may be added over time and as needed.

  • Token counting - Normalized Usage reports what the provider exposes:

    • Anthropic: input vs. cached tokens, output tokens.
    • OpenAI: prompt vs. cached tokens, completion tokens, reasoning tokens where available.

  • Heuristic prompt filtering.

    • ModelParam.MaxPromptLength triggers sdkutil.FilterMessagesByTokenCount, which uses a simple heuristic token counter. It is approximate, not an exact tokenizer.

Development

  • Formatting follows gofumpt and golines via golangci-lint, which is also used for linting. All rules are in .golangci.yml.
  • Useful scripts are defined in taskfile.yml; requires Task.
  • Bug reports and PRs are welcome:
    • Keep the public API (package inference and spec) small and intentional.
    • Avoid leaking provider‑specific types through the public surface; put them under internal/.
    • Please run tests and linters before sending a PR.

License

Copyright (c) 2026 - Present - Pankaj Pipada

All source code in this repository, unless otherwise noted, is licensed under the MIT License. See LICENSE for details.

Documentation

Overview

Package inference provides a single, normalized interface for getting language model completions from multiple providers.

The main entry point is ProviderSetAPI, which lets you:

  • register one or more providers (Anthropic, OpenAI Chat Completions, OpenAI Responses, ...),
  • configure and rotate API keys,
  • send normalized completion requests and receive normalized outputs,
  • optionally stream partial text / reasoning and capture HTTP‑level debug information.

Index

Constants

View Source 
const DataContractHash = "sha256:3c3ccc426b5cec85351f05c8b4fa16c9ad3dfb3cdf0beaa2280d2494cb412f1f"

DataContractHash is a SHA-256 of the contents of DataContractFiles. It is validated by tests and can be used by downstream consumers to check that they are running against the contract version they were built for.

Format: "sha256:<hexstring>".

View Source 
const DataContractVersion = "v1.0.0"

DataContractVersion is bumped when the *schema* of the contract types changes.

Variables

View Source 
var DataContractFiles = []string{
	"spec/data_cache.go",
	"spec/data_citation.go",
	"spec/data_content.go",
	"spec/data_error.go",
	"spec/data_io_union.go",
	"spec/data_model.go",
	"spec/data_tool.go",
}

DataContractFiles lists files that define the data contract. Paths are relative to the repo root.

These files should only contain data-contract types (structs/enums) that downstream consumers rely on structurally. Any change to these files will change the contract hash. It does NOT contain api contracts.

Functions

func ComputeDataContractHash 

func ComputeDataContractHash() (string, error)

ComputeDataContractHash recomputes the SHA-256 hash of the contract files' contents. It is intended for use in tests and development tooling.

NOTE: This function assumes it is run in a source checkout of the module where the paths in DataContractFiles exist on disk. It is not suitable for use in production binaries where the Go source tree might not be available.

func ValidateDataContract 

func ValidateDataContract() error

ValidateDataContract recomputes the hash and compares it to DataContractHash. Tests in this module should call this to enforce that any schema change in the contract files is accompanied by an explicit update of DataContractHash (and, if breaking, DataContractVersion).

Types

type AddProviderConfig 

type AddProviderConfig struct {
	SDKType                  spec.ProviderSDKType `json:"sdkType"`
	Origin                   string               `json:"origin"`
	ChatCompletionPathPrefix string               `json:"chatCompletionPathPrefix"`
	APIKeyHeaderKey          string               `json:"apiKeyHeaderKey"`
	DefaultHeaders           map[string]string    `json:"defaultHeaders"`
}

type DataContractInfo 

type DataContractInfo struct {
	Version string   `json:"version"`
	Hash    string   `json:"hash"`
	Files   []string `json:"files"`
}

DataContractInfo is the public shape returned to callers who want to validate they are compatible with this version of the contract.

func GetDataContractInfo 

func GetDataContractInfo() DataContractInfo

GetDataContractInfo returns the current contract version/hash metadata.

type DebugClientBuilder 

type DebugClientBuilder func(p spec.ProviderParam) spec.CompletionDebugger

DebugClientBuilder constructs a CompletionDebugger for a given provider. A nil builder or a nil returned debugger disable debugging for that provider.

type ProviderSetAPI 

type ProviderSetAPI struct {
	// contains filtered or unexported fields
}

func NewProviderSetAPI 

func NewProviderSetAPI(
	opts ...ProviderSetOption,
) (*ProviderSetAPI, error)

NewProviderSetAPI creates a new ProviderSet and installs the process-wide logger used by this SDK. The logger is chosen via WithLoggerBuilder; if no builder is provided or it returns nil, a no-op logger is used.

func (*ProviderSetAPI) AddProvider 

func (ps *ProviderSetAPI) AddProvider(
	ctx context.Context,
	provider spec.ProviderName,
	config *AddProviderConfig,
) (spec.ProviderParam, error)

func (*ProviderSetAPI) DeleteProvider 

func (ps *ProviderSetAPI) DeleteProvider(
	ctx context.Context,
	provider spec.ProviderName,
) error

func (*ProviderSetAPI) FetchCompletion 

func (ps *ProviderSetAPI) FetchCompletion(
	ctx context.Context,
	provider spec.ProviderName,
	fetchCompletionRequest *spec.FetchCompletionRequest,
	opts *spec.FetchCompletionOptions,
) (*spec.FetchCompletionResponse, error)

FetchCompletion processes a completion request for a given provider.

func (*ProviderSetAPI) SetProviderAPIKey 

func (ps *ProviderSetAPI) SetProviderAPIKey(
	ctx context.Context,
	provider spec.ProviderName,
	apiKey string,
) error

SetProviderAPIKey sets the key for a given provider.

type ProviderSetOption 

type ProviderSetOption func(*ProviderSetAPI)

ProviderSetOption configures optional behavior for ProviderSetAPI.

func WithDebugClientBuilder 

func WithDebugClientBuilder(builder DebugClientBuilder) ProviderSetOption

WithDebugClientBuilder configures a CompletionDebugger factory. The builder is invoked once per provider when it is added. Returning nil disables debugging for that provider.

func WithLogger 

func WithLogger(logger *slog.Logger) ProviderSetOption

type SetProviderAPIKeyRequestBody 

type SetProviderAPIKeyRequestBody struct {
	APIKey string `json:"apiKey" required:"true"`
}

type SetProviderAPIKeyResponse 

type SetProviderAPIKeyResponse struct{}

Source Files

View all Source files

Directories

Path Synopsis
internal
anthropicsdk
integration
Package integration contains example-style tests that demonstrate how to call the inference-go library against real providers.
 Package integration contains example-style tests that demonstrate how to call the inference-go library against real providers.
logutil
openaichatsdk
openairesponsessdk
sdkutil

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.