inference

README

LLM Inference for Go

A single normalized Go interface for LLM inference across multiple providers, using their official SDKs where available.

• Features at a glance
• Installation
• Quickstart
• Examples
• Provider configuration
• Supported providers
  • Anthropic Messages API
  • OpenAI Responses API
  • OpenAI Chat Completions API
  • Google Generate Content API
• Model capabilities and normalization
• HTTP debugging
• Notes
• Development
• License

Features at a glance

• Single normalized interface via ProviderSetAPI

• Provider support today:

  • Anthropic Messages API via github.com/anthropics/anthropic-sdk-go
  • OpenAI Responses API via github.com/openai/openai-go/v3
  • OpenAI Chat Completions API via github.com/openai/openai-go/v3
  • Google Generate Content API via google.golang.org/genai

• Normalized request/response model in spec/:

  • text, image, and file input content
  • assistant/user/tool/reasoning content
  • function/custom/web-search tool definitions and tool calls
  • structured output and verbosity controls
  • reasoning/thinking controls
  • streaming events for text and thinking
  • usage accounting
  • cache-control normalization where supported

• Request normalization before provider calls:

  • capability-driven validation and safe dropping of unsupported features
  • warnings returned in FetchCompletionResponse.Warnings
  • per-model capability override support through FetchCompletionOptions.CapabilityResolver

• Streaming:

  • text streaming for supported providers
  • thinking/reasoning streaming where the provider exposes it

• Debugging:

  • pluggable CompletionDebugger
  • built-in HTTP debugger in debugclient

Installation

# Go 1.26+
go get github.com/flexigpt/inference-go

Quickstart

Basic flow:

1. Create a ProviderSetAPI
2. Register one or more providers with AddProvider
3. Set each provider API key with SetProviderAPIKey
4. Call FetchCompletion

Examples

Available repository examples:

• Anthropic

  • Basic Anthropic call
  • Anthropic tools + streaming + reasoning

• OpenAI

  • Basic OpenAI Chat Completions
  • OpenAI Responses basic
  • OpenAI Responses tools + attachments + streaming
  • OpenAI Chat tools + JSON Schema + streaming

• Google

  • Google Generate Content basic
  • Google Generate Content function-tool round trip
  • Google Generate Content web search + thinking + streaming

• Capability override example (get provider caps, override per-model)

Provider configuration

Providers are registered dynamically with ProviderSetAPI.AddProvider.

type AddProviderConfig struct {
    SDKType                  spec.ProviderSDKType
    Origin                   string
    ChatCompletionPathPrefix string
    APIKeyHeaderKey          string
    DefaultHeaders           map[string]string
}

Fields:

• SDKType

  • spec.ProviderSDKTypeAnthropic
  • spec.ProviderSDKTypeOpenAIChatCompletions
  • spec.ProviderSDKTypeOpenAIResponses
  • spec.ProviderSDKTypeGoogleGenerateContent

• Origin

  • Required
  • Base origin for the provider or gateway/proxy

• ChatCompletionPathPrefix

  • Optional generic path prefix
  • Historical field name, reused across providers
  • Useful when routing through a gateway path prefix
  • Adapters trim built-in endpoint suffixes when needed:
    • Anthropic: trailing v1/messages
    • OpenAI Chat: trailing chat/completions
    • OpenAI Responses: trailing responses

• APIKeyHeaderKey

  • Optional override for non-standard gateway auth headers

• DefaultHeaders

  • Optional extra headers added to every request

Supported providers

Anthropic Messages API

Area	Support	Notes
Text input/output	yes	User/assistant messages normalized
Streaming text	yes
Reasoning/thinking	yes	Signed thinking and redacted thinking supported
Streaming thinking	yes	Redacted thinking is not streamed
Output format	yes	text and jsonSchema
Output verbosity	yes	maps to Anthropic effort
Stop sequences	yes	maps to stop_sequences
Images input	yes	base64 or URL
Files input	partial	PDFs supported; plain-text file document mapping is still pending
Function/custom tools	yes
Web search	yes	server-side web search tool and result blocks
Tool policy	yes	auto, any, tool, none
Cache control	partial	top-level, input/output content, tool choice, tool call, tool output
Citations	partial	URL citations normalized
Usage	yes	input/output/cached; no explicit reasoning token count from Anthropic

Normalization notes:

• reasoning input history keeps Anthropic-compatible signed/redacted reasoning only
• if an interleaved tool-result turn requires Anthropic thinking to be enabled/disabled, the adapter applies the needed override
• tool-result ordering is normalized for Anthropic’s strict tool-use/tool-result turn rules

OpenAI Responses API

Area	Support	Notes
Text input/output	yes
Streaming text	yes
Reasoning/thinking	yes	config + reasoning output items
Streaming thinking	yes	reasoning summary and reasoning text deltas
Output format	yes	text and jsonSchema
Output verbosity	yes
Stop sequences	no	dropped with warning by normalization
Images input	yes	base64 or URL
Files input	yes	base64 or URL
Function/custom tools	yes	custom tool definitions are currently emitted as function tools
Web search	yes	built-in web search tool
Tool policy	yes	auto, any, tool, none
Cache control	partial	top-level prompt cache only
Citations	yes	URL citations normalized
Usage	yes	input/output/cached/reasoning

Normalization notes:

• reasoning input history is sanitized to OpenAI-compatible encrypted reasoning only
• if no encrypted reasoning input exists, reasoning history items are dropped
• stateful Responses features like previous_response_id and provider-side storage are intentionally not normalized

OpenAI Chat Completions API

Area	Support	Notes
Text input/output	yes	first choice only is surfaced
Streaming text	yes
Reasoning config	yes	reasoning effort only
Streaming thinking	no	API does not expose separate reasoning stream
Reasoning message history	no	dropped by adapter
Output format	yes	text and jsonSchema
Output verbosity	yes	max maps to high
Stop sequences	yes	up to 4
Images input	yes	base64 data URL or remote URL
Files input	partial	embedded file data only
Function/custom tools	yes	custom tool definitions are currently emitted as function tools
Web search	yes	via top-level web_search_options, not as a normal tool call
Tool policy	yes	auto, any, tool, none
Cache control	partial	top-level prompt cache only
Citations	yes	URL citations from annotations
Usage	yes	input/output/cached/reasoning
System prompt role	yes	sent as developer for o* / gpt-5* model families, else system

Normalization notes:

• reasoning message inputs are dropped because Chat Completions does not support structured reasoning history
• tool outputs are normalized back in as text-only tool messages
• web search forcing semantics differ from function tools because Chat Completions exposes web search as top-level request options, not as a standard tool call

Google Generate Content API

Area	Support	Notes
Text input/output	yes	first candidate only is surfaced
Streaming text	yes
Reasoning/thinking	yes	config + Google-native signed thought history; signatures on assistant text and function-tool-call parts are preserved for replay
Streaming thinking	yes	streams thought text when exposed by the API
Output format	partial	text and jsonSchema; currently only the raw schema payload is forwarded
Output verbosity	no	dropped with warning by normalization
Stop sequences	yes	normalized up to capability max
Images input	yes	inline bytes or URI
Files input	yes	inline bytes or URI
Function/custom tools	yes	custom tool definitions are emitted as function declarations
Web search	yes	Google Search grounding normalized as web-search call/output
Tool policy	partial	auto, any, tool, none for callable tools; web search cannot be forced as a callable tool
Cache control	no	dropped with warning by normalization
Citations	partial	grounding is normalized as web-search tool outputs, not attached to text citations yet
Usage	yes	input/output/cached/reasoning

Normalization notes:

• reasoning input history keeps only valid Google-native signed thoughts
• non-Google reasoning history is sanitized out before request conversion
• assistant text/tool-call signatures emitted by Gemini are preserved and passed back on follow-up turns
• function tool output history is currently text-only
• ToolPolicy.DisableParallel is not currently normalized for Google Generate Content

Model capabilities and normalization

Capabilities are described by spec.ModelCapabilities in spec/capability.go.

Default provider capability profiles live in:

• Anthropic: internal/anthropicsdk/capability.go
• OpenAI Responses: internal/openairesponsessdk/capability.go
• OpenAI Chat: internal/openaichatsdk/capability.go
• Google Generate Content: internal/googlegeneratecontentsdk/capability.go

You can inspect the active provider-wide default via:

• ProviderSetAPI.GetProviderCapability(ctx, providerName)

Normalization behavior:

• unsupported contract-like features generally return an error
  • example: unsupported output format
• unsupported safe-to-drop features are removed and reported via FetchCompletionResponse.Warnings
  • example: unsupported verbosity or cache-control scope
• some provider-specific history items are sanitized before request conversion
  • Anthropic: only Anthropic-compatible reasoning history is retained
  • OpenAI Responses: only encrypted reasoning history is retained
  • OpenAI Chat: reasoning history is dropped
  • Google: only valid signed Google thought history is retained

For per-model capability differences, pass a custom spec.ModelCapabilityResolver in FetchCompletionOptions.

HTTP debugging

The library exposes a pluggable CompletionDebugger:

type CompletionDebugger interface {
    HTTPClient(base *http.Client) *http.Client
    StartSpan(ctx context.Context, info *spec.CompletionSpanStart) (context.Context, spec.CompletionSpan)
}

Package debugclient includes a ready-to-use implementation:

• wraps provider SDK HTTP clients
• captures scrubbed request/response metadata
• redacts secrets and sensitive content
• attaches structured debug data to FetchCompletionResponse.DebugDetails

Typical setup:

dbg := debugclient.NewHTTPCompletionDebugger(&debugclient.DebugConfig{
    LogToSlog: false,
})

ps, _ := inference.NewProviderSetAPI(
    inference.WithDebugClientBuilder(func(p spec.ProviderParam) spec.CompletionDebugger {
        return dbg
    }),
)

Notes

• Stateless focus

  • the SDK intentionally focuses on stateless request/response flows
  • provider-native conversation state, uploaded file IDs, stored responses, and similar stateful features are out of scope for the normalized interface

• Opaque provider-specific fields

  • many provider-native details remain available only through debug payloads, not the normalized response structs

• Prompt filtering

  • ModelParam.MaxPromptLength uses a heuristic tokenizer via sdkutil.FilterMessagesByTokenCount
  • it is approximate, not a provider tokenizer

• Choice/candidate handling

  • OpenAI Chat surfaces the first choice
  • Google Generate Content surfaces the first candidate

Development

• Formatting/linting uses the repository configuration in .golangci.yml
• Useful scripts are available in taskfile.yml
• PRs are welcome
  • keep the public surface small and provider-neutral
  • avoid leaking provider SDK types into package inference or spec

License

Copyright (c) 2026 - Present - Pankaj Pipada

All source code in this repository, unless otherwise noted, is licensed under the MIT License. See LICENSE for details.

Documentation

Overview

Package inference provides a single, normalized interface for getting language model completions from multiple providers.

The main entry point is ProviderSetAPI, which lets you:

• register one or more providers (Anthropic, OpenAI Chat Completions, OpenAI Responses, ...),
• configure and rotate API keys,
• send normalized completion requests and receive normalized outputs,
• optionally stream partial text / reasoning and capture HTTP‑level debug information.

Index

• Constants
• Variables
• func ComputeDataContractHash() (string, error)
• func ValidateDataContract() error
• type AddProviderConfig
• type DataContractInfo
  • func GetDataContractInfo() DataContractInfo
• type DebugClientBuilder
• type ProviderSetAPI
  • func NewProviderSetAPI(opts ...ProviderSetOption) (*ProviderSetAPI, error)
  • func (ps *ProviderSetAPI) AddProvider(ctx context.Context, provider spec.ProviderName, config *AddProviderConfig) (spec.ProviderParam, error)
  • func (ps *ProviderSetAPI) DeleteProvider(ctx context.Context, provider spec.ProviderName) error
  • func (ps *ProviderSetAPI) FetchCompletion(ctx context.Context, provider spec.ProviderName, ...) (*spec.FetchCompletionResponse, error)
  • func (ps *ProviderSetAPI) GetProviderCapability(ctx context.Context, provider spec.ProviderName) (spec.ModelCapabilities, error)
  • func (ps *ProviderSetAPI) SetProviderAPIKey(ctx context.Context, provider spec.ProviderName, apiKey string) error
• type ProviderSetOption
  • func WithDebugClientBuilder(builder DebugClientBuilder) ProviderSetOption
  • func WithLogger(logger *slog.Logger) ProviderSetOption
• type SetProviderAPIKeyRequestBody
• type SetProviderAPIKeyResponse

Constants

const DataContractHash = "sha256:40d4d92460554cc5959464baa982f4cb6178fe77639d6c6bc01c6e682a9bd1ee"

DataContractHash is a SHA-256 of the contents of DataContractFiles. It is validated by tests and can be used by downstream consumers to check that they are running against the contract version they were built for.

Format: "sha256:<hexstring>".

const DataContractVersion = "v1.0.0"

DataContractVersion is bumped when the *schema* of the contract types changes.

Variables

var DataContractFiles = []string{
	"spec/data_cache.go",
	"spec/data_citation.go",
	"spec/data_content.go",
	"spec/data_error.go",
	"spec/data_io_union.go",
	"spec/data_model.go",
	"spec/data_tool.go",
}

DataContractFiles lists files that define the data contract. Paths are relative to the repo root.

These files should only contain data-contract types (structs/enums) that downstream consumers rely on structurally. Any change to these files will change the contract hash. It does NOT contain api contracts.

Functions

func ComputeDataContractHash

func ComputeDataContractHash() (string, error)

ComputeDataContractHash recomputes the SHA-256 hash of the contract files' contents. It is intended for use in tests and development tooling.

NOTE: This function assumes it is run in a source checkout of the module where the paths in DataContractFiles exist on disk. It is not suitable for use in production binaries where the Go source tree might not be available.

func ValidateDataContract

func ValidateDataContract() error

ValidateDataContract recomputes the hash and compares it to DataContractHash. Tests in this module should call this to enforce that any schema change in the contract files is accompanied by an explicit update of DataContractHash (and, if breaking, DataContractVersion).

Types

type AddProviderConfig

type AddProviderConfig struct {
	SDKType                  spec.ProviderSDKType `json:"sdkType"`
	Origin                   string               `json:"origin"`
	ChatCompletionPathPrefix string               `json:"chatCompletionPathPrefix"`
	APIKeyHeaderKey          string               `json:"apiKeyHeaderKey"`
	DefaultHeaders           map[string]string    `json:"defaultHeaders"`
}

type DataContractInfo

type DataContractInfo struct {
	Version string   `json:"version"`
	Hash    string   `json:"hash"`
	Files   []string `json:"files"`
}

DataContractInfo is the public shape returned to callers who want to validate they are compatible with this version of the contract.

func GetDataContractInfo

func GetDataContractInfo() DataContractInfo

GetDataContractInfo returns the current contract version/hash metadata.

type DebugClientBuilder

type DebugClientBuilder func(p spec.ProviderParam) spec.CompletionDebugger

DebugClientBuilder constructs a CompletionDebugger for a given provider. A nil builder or a nil returned debugger disable debugging for that provider.

type ProviderSetAPI

type ProviderSetAPI struct {
	// contains filtered or unexported fields
}

func NewProviderSetAPI

func NewProviderSetAPI(
	opts ...ProviderSetOption,
) (*ProviderSetAPI, error)

NewProviderSetAPI creates a new ProviderSet and installs the process-wide logger used by this SDK. The logger is chosen via WithLoggerBuilder; if no builder is provided or it returns nil, a no-op logger is used.

func (*ProviderSetAPI) AddProvider

func (ps *ProviderSetAPI) AddProvider(
	ctx context.Context,
	provider spec.ProviderName,
	config *AddProviderConfig,
) (spec.ProviderParam, error)

func (*ProviderSetAPI) DeleteProvider

func (ps *ProviderSetAPI) DeleteProvider(
	ctx context.Context,
	provider spec.ProviderName,
) error

func (*ProviderSetAPI) FetchCompletion

func (ps *ProviderSetAPI) FetchCompletion(
	ctx context.Context,
	provider spec.ProviderName,
	fetchCompletionRequest *spec.FetchCompletionRequest,
	opts *spec.FetchCompletionOptions,
) (*spec.FetchCompletionResponse, error)

FetchCompletion processes a completion request for a given provider.

func (*ProviderSetAPI) GetProviderCapability

func (ps *ProviderSetAPI) GetProviderCapability(
	ctx context.Context,
	provider spec.ProviderName,
) (spec.ModelCapabilities, error)

func (*ProviderSetAPI) SetProviderAPIKey

func (ps *ProviderSetAPI) SetProviderAPIKey(
	ctx context.Context,
	provider spec.ProviderName,
	apiKey string,
) error

SetProviderAPIKey sets the key for a given provider.

type ProviderSetOption

type ProviderSetOption func(*ProviderSetAPI)

ProviderSetOption configures optional behavior for ProviderSetAPI.

func WithDebugClientBuilder

func WithDebugClientBuilder(builder DebugClientBuilder) ProviderSetOption

WithDebugClientBuilder configures a CompletionDebugger factory. The builder is invoked once per provider when it is added. Returning nil disables debugging for that provider.

func WithLogger

func WithLogger(logger *slog.Logger) ProviderSetOption

type SetProviderAPIKeyRequestBody

type SetProviderAPIKeyRequestBody struct {
	APIKey string `json:"apiKey" required:"true"`
}

type SetProviderAPIKeyResponse

type SetProviderAPIKeyResponse struct{}