llmgateway

package module
v0.1.0-dev.12 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 31, 2026 License: LGPL-3.0-or-later Imports: 21 Imported by: 0

README

llmgateway

llmgateway is an embeddable Go engine for authenticated, policy-controlled, multi-provider LLM inference. It owns protocol normalization, routing, retries, stream commit semantics, guardrails, caching decisions, and accounting state transitions. Hosts provide persistence, identity, secrets, distributed counters, and telemetry through explicit interfaces.

The module intentionally has no dependency on Daptin, Gin, api2go, SQL, database drivers, or distributed-cache clients. Daptin consumes tagged releases and supplies those host integrations from its own repository.

The compatibility surface is declared in compatibility/manifest.json. An endpoint or provider is supported only when its manifest entry is backed by its conformance suite.

The verified provider surface is currently the strict OpenAI-compatible adapter. Named services are certified separately against the live-provider matrix before they are added to the manifest; the presence of a compatible URL alone is not treated as certification.

Development

go test ./...
go test -race ./...

The example in examples/basic proves that the catalog and lifecycle can be embedded by a service that has no Daptin dependency.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrDraining     = errors.New("llmgateway is draining")
	ErrNotReady     = errors.New("llmgateway has no valid catalog snapshot")
	ErrCounterLimit = errors.New("llmgateway transient counter limit exceeded")
)

Functions

This section is empty.

Types

type AccountingStore 

type AccountingStore interface {
	Admit(context.Context, contract.Admission) (contract.ReservationToken, error)
	Finalize(context.Context, contract.Completion) error
	Cancel(context.Context, contract.Cancellation) error
	ReapExpired(context.Context, time.Time, int) (contract.ReapResult, error)
}

type Authenticator 

type Authenticator interface {
	Authenticate(context.Context, string) (contract.Principal, error)
}

type AuthenticatorFunc 

type AuthenticatorFunc func(context.Context, string) (contract.Principal, error)

func (AuthenticatorFunc) Authenticate 

func (f AuthenticatorFunc) Authenticate(ctx context.Context, credential string) (contract.Principal, error)

type Authorizer 

type Authorizer interface {
	Authorize(context.Context, contract.Principal, catalog.Model) error
}

type CatalogSource 

type CatalogSource interface {
	Load(context.Context, uint64) (catalog.Document, error)
}

type Clock 

type Clock interface {
	Now() time.Time
	NewTimer(time.Duration) Timer
}

type CounterStore 

type CounterStore interface {
	Add(context.Context, string, int64, time.Duration) (int64, error)
	Get(context.Context, string) (int64, bool, error)
	Acquire(context.Context, string, int64, time.Duration) (string, error)
	Release(context.Context, string) error
	Delete(context.Context, string) error
}

type Dependencies 

type Dependencies struct {
	Catalog    CatalogSource
	Secrets    SecretResolver
	Adapters   *adapter.Registry
	Authorizer Authorizer
	Accounting AccountingStore
	Counters   CounterStore
	Cache      ResponseCache
	Guardrails *guardrail.Registry
	Telemetry  TelemetrySink
	Selector   Selector
	Clock      Clock
}

type DisabledResponseCache 

type DisabledResponseCache struct{}

DisabledResponseCache is an explicit cache opt-out.

func (DisabledResponseCache) Delete 

func (DisabledResponseCache) Delete(context.Context, string) error

func (DisabledResponseCache) Get 

func (DisabledResponseCache) Get(context.Context, string) ([]byte, bool, error)

func (DisabledResponseCache) Set 

func (DisabledResponseCache) Set(context.Context, string, []byte, time.Duration) error

type DiscardTelemetrySink 

type DiscardTelemetrySink struct{}

DiscardTelemetrySink is an explicit opt-out for hosts that do not emit gateway telemetry.

func (DiscardTelemetrySink) Record 

func (DiscardTelemetrySink) Record(context.Context, TelemetryEvent)

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

func New 

func New(dependencies Dependencies, options Options) (*Engine, error)

func (*Engine) Authorize 

func (e *Engine) Authorize(ctx context.Context, principal contract.Principal, publicModel string) error

Authorize evaluates model visibility through the same host policy used by invocation. Protocol handlers use it for discovery without duplicating ACLs.

func (*Engine) Drain 

func (e *Engine) Drain(ctx context.Context) error

func (*Engine) Handler 

func (e *Engine) Handler(options HTTPOptions) (http.Handler, error)

Handler exposes the engine's supported OpenAI-compatible HTTP routes.

func (*Engine) Invoke 

func (e *Engine) Invoke(ctx context.Context, principal contract.Principal, request contract.Request) (contract.Response, error)

func (*Engine) Probe 

func (e *Engine) Probe(ctx context.Context) (HealthReport, error)

Probe executes only explicitly enabled, side-effect-free adapter health checks. Successful probes clear infrastructure circuit state but never clear provider rate-limit cooldowns.

func (*Engine) Reload 

func (e *Engine) Reload(ctx context.Context) error

func (*Engine) Snapshot 

func (e *Engine) Snapshot() (*catalog.Snapshot, error)

func (*Engine) Status 

func (e *Engine) Status() Status

func (*Engine) Stream 

func (e *Engine) Stream(ctx context.Context, principal contract.Principal, request contract.Request) (EventStream, error)

type EventStream 

type EventStream = contract.EventStream

type FallbackCounterStore 

type FallbackCounterStore struct {
	// contains filtered or unexported fields
}

FallbackCounterStore keeps disposable deployment protection available when distributed coordination fails. Durable budgets remain the responsibility of AccountingStore and never use this fallback.

func NewFallbackCounterStore 

func NewFallbackCounterStore(primary, fallback CounterStore, onError func(error)) (*FallbackCounterStore, error)

func (FallbackCounterStore) Acquire 

func (s FallbackCounterStore) Acquire(ctx context.Context, key string, maximum int64, ttl time.Duration) (string, error)

func (FallbackCounterStore) Add 

func (s FallbackCounterStore) Add(ctx context.Context, key string, delta int64, ttl time.Duration) (int64, error)

func (FallbackCounterStore) Delete 

func (s FallbackCounterStore) Delete(ctx context.Context, key string) error

func (FallbackCounterStore) Get 

func (s FallbackCounterStore) Get(ctx context.Context, key string) (int64, bool, error)

func (FallbackCounterStore) Release 

func (s FallbackCounterStore) Release(ctx context.Context, token string) error

type GatewayStream 

type GatewayStream struct {
	// contains filtered or unexported fields
}

func (*GatewayStream) Close 

func (s *GatewayStream) Close(ctx context.Context) error

func (*GatewayStream) Next 

func (s *GatewayStream) Next(ctx context.Context) (contract.StreamEvent, error)

Next hands the first validated event to the caller and irrevocably commits this stream to its selected deployment. Provider switching is impossible after this method returns an event.

type HTTPOptions 

type HTTPOptions struct {
	Authenticator Authenticator
	Protocol      openai.Options
}

HTTPOptions configures the strict OpenAI-compatible HTTP surface. The authenticator is supplied here so hosts that invoke the engine directly do not need to configure an unused HTTP authentication path.

type HealthReport 

type HealthReport struct {
	Checked   int
	Healthy   int
	Unhealthy int
}

type LocalCounterStore 

type LocalCounterStore struct {
	// contains filtered or unexported fields
}

LocalCounterStore is a bounded-process fallback for transient counters and leases. Expired entries are discarded opportunistically on access.

func NewLocalCounterStore 

func NewLocalCounterStore() *LocalCounterStore

func (*LocalCounterStore) Acquire 

func (s *LocalCounterStore) Acquire(ctx context.Context, key string, maximum int64, ttl time.Duration) (string, error)

func (*LocalCounterStore) Add 

func (s *LocalCounterStore) Add(ctx context.Context, key string, delta int64, ttl time.Duration) (int64, error)

func (*LocalCounterStore) Delete 

func (s *LocalCounterStore) Delete(ctx context.Context, key string) error

func (*LocalCounterStore) Get 

func (s *LocalCounterStore) Get(ctx context.Context, key string) (int64, bool, error)

func (*LocalCounterStore) Release 

func (s *LocalCounterStore) Release(ctx context.Context, token string) error

type Options 

type Options struct {
	MaxAttempts         int
	FinalizationTimeout time.Duration
	BaseRetryDelay      time.Duration
	MaxRetryDelay       time.Duration
	CircuitFailures     int64
	CircuitWindow       time.Duration
	CircuitCooldown     time.Duration
	CacheTTL            time.Duration
	CacheTimeout        time.Duration
	MaxCacheEntryBytes  int
	FirstEventTimeout   time.Duration
	StreamIdleTimeout   time.Duration
	RequestTimeout      time.Duration
	HealthProbeTimeout  time.Duration
	HealthProbeWorkers  int
}

type RandomSelector 

type RandomSelector struct{}

RandomSelector is the production weighted-route selector. Tests should use an injected deterministic selector instead.

func (RandomSelector) Intn 

func (RandomSelector) Intn(limit int) int

type ResponseCache 

type ResponseCache interface {
	Get(context.Context, string) ([]byte, bool, error)
	Set(context.Context, string, []byte, time.Duration) error
	Delete(context.Context, string) error
}

type SecretResolver 

type SecretResolver interface {
	ResolveSecret(context.Context, string) ([]byte, error)
}

type Selector 

type Selector interface {
	Intn(int) int
}

type Status 

type Status struct {
	Ready            bool   `json:"ready"`
	Draining         bool   `json:"draining"`
	Degraded         bool   `json:"degraded"`
	Revision         uint64 `json:"revision"`
	RejectedRevision uint64 `json:"rejected_revision,omitempty"`
	ReloadStage      string `json:"reload_stage,omitempty"`
}

Status is a redacted, point-in-time view of engine lifecycle and catalog readiness. Failure stages are stable categories and never contain host or provider error text.

type SystemClock 

type SystemClock struct{}

SystemClock is the production wall clock implementation.

func (SystemClock) NewTimer 

func (SystemClock) NewTimer(duration time.Duration) Timer

func (SystemClock) Now 

func (SystemClock) Now() time.Time

type TelemetryEvent 

type TelemetryEvent struct {
	Name       string
	RequestID  contract.ID
	Revision   uint64
	Attributes map[string]string
	Measures   map[string]int64
}

type TelemetrySink 

type TelemetrySink interface {
	Record(context.Context, TelemetryEvent)
}

type Timer 

type Timer interface {
	C() <-chan time.Time
	Stop() bool
}

Source Files

View all Source files

Directories

Path Synopsis
openaicompat
Package openaicompat implements a strict OpenAI-compatible upstream adapter.
 Package openaicompat implements a strict OpenAI-compatible upstream adapter.
examples
basic command
internal
routing
protocol
openai

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.