aigateway

README

togo-framework/ai-gateway

An AI gateway for togo — rate limits, spend caps, caching, prompt management & provider fallback.

Install

togo install togo-framework/ai-gateway

A control plane in front of LLM providers (Cortex-style). Wrap the togo AI kit with per-key rate limits + spend caps, response caching, prompt-template management, and provider routing/fallback so you can expose AI safely to many consumers and bill it.

Quick start

g, _ := aigateway.FromKernel(k)

// Bridge your togo `ai` drivers to the gateway (order = routing priority).
aigateway.RegisterProvider(myOpenAIAdapter)
aigateway.RegisterProvider(myAnthropicAdapter) // fallback if the first errors

// Mint a metered key.
raw, _ := g.CreateKey("mobile-app", aigateway.Key{
    AllowedModels: []string{"gpt-4o-mini"},
    RatePerMin:    60,
    MonthlyCap:    50.00, // USD
})

resp, err := g.Complete(ctx, raw, aigateway.Request{
    Model:    "gpt-4o-mini",
    Messages: []aigateway.Message{{Role: "user", Content: "Hi"}},
})

Features

• Per-key rate limiting — sliding window, requests/min (429 over limit).
• Spend caps — monthly + hard caps enforced before the provider call (402 over quota), with a per-model price book and accumulated spend.
• Response caching — SHA-256 over (model | messages | temperature | max_tokens), TTL (default 1h); cache hits are returned instantly and billed at zero.
• Prompt management — named, versioned templates with {{var}} interpolation (RenderPrompt), activate any version.
• Provider routing + fallback — tries registered providers in order; falls back to the next on error.
• API keys — hashed (SHA-256), per-key allowed-models + limits, revocation; raw key shown once.

Provider interface

The gateway is decoupled from any SDK — implement a tiny adapter over your togo ai driver:

type Provider interface {
    Name() string
    Complete(ctx context.Context, req aigateway.Request) (aigateway.Response, error)
}

REST API

/complete reads the key from Authorization: Bearer <key>.

Method	Path	Description
POST	/api/ai-gateway/complete	gated completion
GET/POST/DELETE	/api/ai-gateway/keys[/{id}]	API-key management
GET/POST	/api/ai-gateway/prompts[/{name}]	prompt templates + versions
GET	/api/ai-gateway/usage?key=	usage/spend records

Configuration

No required env. Set per-model prices with g.SetPrice(model, aigateway.Price{Input, Output}); default cache TTL is 1h. Swap the in-memory stores via the exported seams for DB persistence.

---

Premium sponsors

ID8 Media  ·  One Studio

_{Support togo — become a sponsor.}

Documentation

Overview

Package aigateway is an AI gateway for togo — a control plane in front of LLM providers (Cortex-style). It wraps completions with per-key rate limiting and spend caps, response caching, prompt-template management, and provider routing/fallback, so the AI kit can be exposed safely to many consumers.

Providers implement the small Provider interface (an adapter bridges the togo `ai` plugin's drivers), so the gateway stays decoupled and testable offline.

Index

• Variables
• func RegisterProvider(p Provider)
• type Gateway
  • func FromKernel(k *togo.Kernel) (*Gateway, bool)
  • func New() *Gateway
  • func (g *Gateway) Complete(ctx context.Context, rawKey string, req Request) (Response, error)
  • func (g *Gateway) CreateKey(name string, k Key) (raw string, rec *Key)
  • func (g *Gateway) Keys() []*Key
  • func (g *Gateway) Prompts() *promptStore
  • func (g *Gateway) RenderPrompt(name string, vars map[string]string) (string, bool)
  • func (g *Gateway) Revoke(id string) bool
  • func (g *Gateway) SetPrice(model string, p Price)
  • func (g *Gateway) Usage(keyID string) []UsageRecord
• type Key
• type Message
• type Price
• type Provider
• type Request
• type Response
• type UsageRecord

Variables

var (
	ErrKeyInvalid      = errors.New("ai-gateway: invalid or revoked API key")
	ErrRateLimited     = errors.New("ai-gateway: rate limit exceeded")
	ErrQuotaExceeded   = errors.New("ai-gateway: spend quota exceeded")
	ErrModelNotAllowed = errors.New("ai-gateway: model not allowed for this key")
	ErrNoProvider      = errors.New("ai-gateway: no provider could handle the request")
)

Errors returned by the gateway.

Functions

func RegisterProvider

func RegisterProvider(p Provider)

RegisterProvider adds a provider to the routing chain (order = priority).

Types

type Gateway

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway is the runtime stored on the kernel (k.Get("ai-gateway")).

func FromKernel

func FromKernel(k *togo.Kernel) (*Gateway, bool)

FromKernel returns the gateway.

func New

func New() *Gateway

New builds a Gateway with default pricing + a 1-hour cache.

func (*Gateway) Complete

func (g *Gateway) Complete(ctx context.Context, rawKey string, req Request) (Response, error)

Complete is the gateway entrypoint: authenticate the key, enforce rate limit + spend caps, serve from cache, route to a provider (with fallback), bill, cache.

func (*Gateway) CreateKey

func (g *Gateway) CreateKey(name string, k Key) (raw string, rec *Key)

CreateKey mints an API key, returning the raw key once (only its hash is kept).

func (*Gateway) Keys

func (g *Gateway) Keys() []*Key

Keys lists the API keys (without secrets).

func (*Gateway) Prompts

func (g *Gateway) Prompts() *promptStore

Prompts exposes the prompt-template store.

func (*Gateway) RenderPrompt

func (g *Gateway) RenderPrompt(name string, vars map[string]string) (string, bool)

RenderPrompt renders a stored template's active version with vars.

func (*Gateway) Revoke

func (g *Gateway) Revoke(id string) bool

Revoke disables a key by id.

func (*Gateway) SetPrice

func (g *Gateway) SetPrice(model string, p Price)

SetPrice sets the per-1K-token price for a model.

func (*Gateway) Usage

func (g *Gateway) Usage(keyID string) []UsageRecord

Usage returns recorded usage, optionally filtered by key id.

type Key

type Key struct {
	ID   string `json:"id"`
	Name string `json:"name"`

	AllowedModels []string  `json:"allowed_models,omitempty"`
	RatePerMin    int       `json:"rate_per_min"` // 0 = unlimited
	MonthlyCap    float64   `json:"monthly_cap"`  // USD, 0 = unlimited
	HardCap       float64   `json:"hard_cap"`
	Spend         float64   `json:"spend"`
	CreatedAt     time.Time `json:"created_at"`
	Revoked       bool      `json:"revoked"`
	// contains filtered or unexported fields
}

Key is an API key with per-key limits + accumulated spend.

type Message

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Message is one chat message.

type Price

type Price struct{ Input, Output float64 }

Price is the per-1K-token cost for a model.

type Provider

type Provider interface {
	Name() string
	Complete(ctx context.Context, req Request) (Response, error)
}

Provider is implemented by an LLM backend (or an adapter over the togo ai kit).

type Request

type Request struct {
	Model       string    `json:"model"`
	Messages    []Message `json:"messages"`
	Temperature float64   `json:"temperature,omitempty"`
	MaxTokens   int       `json:"max_tokens,omitempty"`
}

Request is a completion request.

type Response

type Response struct {
	Content      string  `json:"content"`
	Model        string  `json:"model"`
	Provider     string  `json:"provider,omitempty"`
	InputTokens  int     `json:"input_tokens"`
	OutputTokens int     `json:"output_tokens"`
	Cached       bool    `json:"cached"`
	Cost         float64 `json:"cost"`
}

Response is a completion result.

type UsageRecord

type UsageRecord struct {
	KeyID  string    `json:"key_id"`
	Model  string    `json:"model"`
	Input  int       `json:"input_tokens"`
	Output int       `json:"output_tokens"`
	Cost   float64   `json:"cost"`
	Cached bool      `json:"cached"`
	At     time.Time `json:"at"`
}

UsageRecord is one billed request.