llm-router

command module
v0.3.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 1, 2026 License: MIT Imports: 1 Imported by: 0

README

llm-router

Single-binary, bloat-free, extensible multi-provider OpenAI-compatible LLM gateway. Go on the backend, Svelte on the front, compiled into one file.

llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Zero external dependencies.


What It Does

Sits between your applications and LLM providers, exposing a single OpenAI-compatible /v1 API.

  • Load distribution: Split traffic across multiple providers and provider accounts.
  • Access control: Issue separate API keys with individual model, provider, and rate limits.
  • Built-in dashboard: Full admin UI embedded into the go binary.
  • Plugin system: Lua plugins allow you to add new provider integrations easily.

Architecture & Philosophy

  • Blazingly fast: Go + Svelte, no heavy JS runtime.
  • Modular: Core repo remains lightweight and maintainable. Easily extensible with Lua plugins.
  • Everything works: Zero dead code, all features are tested by real people.
  • Clean UI: Fully reviewed Svelte dashboard polished by a human.
  • UX with common sense: No unmoderated AI slop that nobody knows how to use.

Key Features

Admin Panel
  • Chat: Test models directly in the browser.
  • Metrics: Inspect model and provider usage.
  • Providers: Connect new providers and accounts.
  • Models: Browse available models across active providers.
  • Plugins: Install plugins from the online store.
  • Virtual models: Create a composite model out of many.
  • Tokens: Manage access tokens with fine-grained model permissions.
Zero-Config

No config files. Minimal set of optional CLI flags and you are done.

llm-router --help

llm-router — minimalist LLM routing gateway

Usage:
  llm-router [host] [flags]

Examples:
  llm-router
llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Flags:
      --api Port             port for /v1 OpenAI-compatible API (default 8081)
      --db string            path to the database file (default "llm-router.db")
  -h, --help                 help for llm-router
      --log-level LogLevel   log level: debug, info, warn, error (default info)
      --no-auth              disable all authorization (dev and AI debugging only)
  -v, --version              print version information and exit
      --web Port             port for dashboard UI (default 8080)

Quick Start

Development server
make start
make start HOST=localhost WEB_PORT=8080 API_PORT=8081
Production builds
make publish
make publish PUBLISH_PLATFORMS="windows/amd64 linux/amd64 darwin/amd64 freebsd/amd64"
Dashboard setup
  1. Create the initial admin account.
  2. Go to Providers and authenticate in any of listed providers, or add a new one.
  3. Go to Tokens and issue a new token. In wizard, select Allow all if you don't care about limits.
  4. The token will be shown to you once, afterwards you won't be able see it again.
API usage

Pass your token and the endpoint URL to your harness or any other AI-powered software:

  • Base URL: http://localhost:8081/v1
  • Token: 10b7****************************************************4fd7

License

MIT

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Package cmd defines the llm-router CLI using cobra.
Package cmd defines the llm-router CLI using cobra.
internal
adapters/generic
Package generic provides a generic OpenAI-compatible backend for custom providers.
Package generic provides a generic OpenAI-compatible backend for custom providers.
api/v1
Package v1 implements the OpenAI-compatible /v1/...
Package v1 implements the OpenAI-compatible /v1/...
config
Package config holds the runtime configuration populated from CLI flags.
Package config holds the runtime configuration populated from CLI flags.
dashboard
Package dashboard serves the admin SPA and its backing JSON API.
Package dashboard serves the admin SPA and its backing JSON API.
db
Package db manages the embedded bbolt database and all bucket definitions.
Package db manages the embedded bbolt database and all bucket definitions.
errors
Package errors defines standard error types used across llm-router services.
Package errors defines standard error types used across llm-router services.
httpkit
Package httpkit holds shared HTTP transport helpers for the API surfaces.
Package httpkit holds shared HTTP transport helpers for the API surfaces.
logging
Package logging provides the bracket log handler for llm-router: [LEVEL][dd.mm.yyyy h:m:s.ms][key: value] message.
Package logging provides the bracket log handler for llm-router: [LEVEL][dd.mm.yyyy h:m:s.ms][key: value] message.
models
Package models defines all shared data structures for llm-router.
Package models defines all shared data structures for llm-router.
pool
Package pool runs the single-pass credential failover shared by every backend: each key is tried at most once in pool order, the first success wins, otherwise the last error is returned.
Package pool runs the single-pass credential failover shared by every backend: each key is tried at most once in pool order, the first success wins, otherwise the last error is returned.
repository
Package repository provides generic type-safe database operations using Go generics.
Package repository provides generic type-safe database operations using Go generics.
server
Package server wires all services together and starts the HTTP servers.
Package server wires all services together and starts the HTTP servers.
services/admin
Package admin manages the bootstrap flow and dashboard authentication.
Package admin manages the bootstrap flow and dashboard authentication.
services/config
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
services/credential
Package credential implements the Credential Pool Service.
Package credential implements the Credential Pool Service.
services/exhausted
Package exhausted owns the unified joint limit-key store (0.1.1).
Package exhausted owns the unified joint limit-key store (0.1.1).
services/geoban
Package geoban owns indefinite geo-block flags: (plugin, provider type, proxy) triples whose exit the upstream refuses for that provider.
Package geoban owns indefinite geo-block flags: (plugin, provider type, proxy) triples whose exit the upstream refuses for that provider.
services/maintenance
Package maintenance implements the Provider Maintenance & Rotation Service.
Package maintenance implements the Provider Maintenance & Rotation Service.
services/metrics
Package metrics implements the Metrics Service for tracking API usage.
Package metrics implements the Metrics Service for tracking API usage.
services/modelinfo
Package modelinfo manages model metadata caching.
Package modelinfo manages model metadata caching.
services/pluginrepo
Package pluginrepo manages external plugin repositories (the plugin store).
Package pluginrepo manages external plugin repositories (the plugin store).
services/provider
Package provider owns the unified provider registry.
Package provider owns the unified provider registry.
services/proxypool
Package proxypool adapts the proxypool library to router policy and storage.
Package proxypool adapts the proxypool library to router policy and storage.
services/router
Package router implements the Router Service.
Package router implements the Router Service.
services/token
Package token implements the Internal Token Service.
Package token implements the Internal Token Service.
services/virtual
Package virtual implements the Virtual Models service.
Package virtual implements the Virtual Models service.
streamgate
Package streamgate tracks whether a streaming response committed its first byte to the client.
Package streamgate tracks whether a streaming response committed its first byte to the client.
testutil
Package testutil provides shared test utilities for llm-router tests.
Package testutil provides shared test utilities for llm-router tests.
util
Package util provides shared utility functions for the llm-router framework.
Package util provides shared utility functions for the llm-router framework.
providers
virtual
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL