gateway

Documentation

Overview

Package gateway is the HTTP server: authentication with virtual keys, budgets and rate limits, fair-share priorities, routing to vLLM, streaming pass-through, accounting, metrics and the admin API.

Index

• Constants
• func AuditEntryJSON(b []byte) (store.AuditEntry, error)
• func CheckpointKey(seq int64) string
• func ClientAddr(r *http.Request, trusted []netip.Prefix) (netip.Addr, bool)
• func NewestAnchor(ctx context.Context, sink export.Sink) (store.AuditCheckpoint, error)
• func ParseDay(v string) (time.Time, error)
• func ParseGroupBy(v string) ([]string, error)
• func WriteReport(w io.Writer, format string, from, to time.Time, dims []string, ...) error
• type Option
  • func WithClock(now func() time.Time) Option
  • func WithJWTClient(c *http.Client) Option
  • func WithLogger(l *slog.Logger) Option
  • func WithSecretWriter(w SecretWriter) Option
• type SecretWriter
• type Secrets
• type Server
  • func New(cfg *config.Config, st store.Store, secrets Secrets, opts ...Option) (*Server, error)
  • func (s *Server) Close(ctx context.Context)
  • func (s *Server) HTTPServer() *http.Server
  • func (s *Server) Handler() http.Handler
  • func (s *Server) Start(ctx context.Context) error

Constants

const (
	FormatJSON = "json"
	FormatCSV  = "csv"
	// FormatParsable2 is '|'-delimited without a trailing '|', like Slurm
	// sreport --parsable2, for side-by-side reconciliation.
	FormatParsable2 = "parsable2"
)

Report formats.

Functions

func AuditEntryJSON

func AuditEntryJSON(b []byte) (store.AuditEntry, error)

AuditEntryJSON decodes one exported entry back (for verifying an export offline and for tests).

func CheckpointKey

func CheckpointKey(seq int64) string

CheckpointKey is the object name of the checkpoint at seq (zero-padded, so the lexicographic listing is in seq order).

func ClientAddr

func ClientAddr(r *http.Request, trusted []netip.Prefix) (netip.Addr, bool)

ClientAddr determines the client of a request. X-Forwarded-For is read only when the connection comes from a trusted proxy, and then from the right: the first address that is not a trusted proxy is the client (MDN, X-Forwarded-For, "Trusted proxy list"). Leftmost entries can be written by anybody and are never used for security. An entry that does not parse stops the walk at the last trusted hop, which then stands for the client and fails any allowlist meant for real clients.

func NewestAnchor

func NewestAnchor(ctx context.Context, sink export.Sink) (store.AuditCheckpoint, error)

NewestAnchor reads the newest checkpoint an export holds.

func ParseDay

func ParseDay(v string) (time.Time, error)

ParseDay accepts RFC 3339 or YYYY-MM-DD (UTC midnight).

func ParseGroupBy

func ParseGroupBy(v string) ([]string, error)

ParseGroupBy splits a comma-separated dimension list.

func WriteReport

func WriteReport(w io.Writer, format string, from, to time.Time, dims []string, rows []store.ReportRow) error

WriteReport is writeReport for the CLI.

Types

type Option

type Option func(*Server)

Option tunes a Server (tests).

func WithClock

func WithClock(now func() time.Time) Option

WithClock replaces time.Now.

func WithJWTClient

func WithJWTClient(c *http.Client) Option

WithJWTClient sets the HTTP client used for OIDC discovery and the key set (a private CA, a proxy).

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger (default: discard).

func WithSecretWriter

func WithSecretWriter(w SecretWriter) Option

WithSecretWriter sets where scheduled rotations deliver new secrets.

type SecretWriter

type SecretWriter interface {
	Write(ctx context.Context, ref secret.Ref, value string) error
}

SecretWriter writes a secret to a vault reference (secret.Resolver).

type Secrets

type Secrets struct {
	// MasterKey protects the admin API.
	MasterKey string
	// UpstreamKeys maps an upstream key_file path to the vLLM --api-key
	// it holds.
	UpstreamKeys map[string]string
	// MetricsToken, when set, is required as a bearer token on /metrics.
	MetricsToken string
	// CacheSaltKey is the HMAC key the per-tenant cache_salt values are
	// derived from (at least 32 bytes; unused with cache_isolation none).
	CacheSaltKey string
	// WebhookToken and SMTPPassword authenticate soft-budget alerts.
	WebhookToken string
	SMTPPassword string
	// SSOClientSecret is the admin login client's secret (empty: a public
	// client).
	SSOClientSecret string
	// SCIMToken is the bearer token of the provisioning client.
	SCIMToken string
	// AuditKey is the audit checkpoint signing key (an Ed25519 seed, base64
	// or hex); empty: checkpoints are not signed.
	AuditKey string
	// ExportCredentials are the S3 keys of each export, by name.
	ExportCredentials map[string]export.Credentials
	// GuardrailTokens are the bearer tokens of http guardrails, by name.
	GuardrailTokens map[string]string
	// TracingToken is the OTLP collector's bearer token (empty: none).
	TracingToken string
}

Secrets are read from files by the caller and handed over here; the gateway never reads the environment.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is the gateway.

func New

func New(cfg *config.Config, st store.Store, secrets Secrets, opts ...Option) (*Server, error)

New builds a server. Call Start to run its background loops and Close to stop them.

func (*Server) Close

func (s *Server) Close(ctx context.Context)

Close stops the background loops and drains the accounting queue.

func (*Server) HTTPServer

func (s *Server) HTTPServer() *http.Server

HTTPServer returns an http.Server with the configured limits.

func (*Server) Handler

func (s *Server) Handler() http.Handler

Handler returns the HTTP handler.

func (*Server) Start

func (s *Server) Start(ctx context.Context) error

Start ensures configured teams exist, loads budgets and starts the health, budget and accounting loops.