llm-router

command module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 19, 2026 License: MIT Imports: 1 Imported by: 0

README

llm-router

Single-binary, bloat-free, extensible multi-provider OpenAI-compatible LLM gateway. Go on the backend, Svelte on the front, compiled into one file.

llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Zero external dependencies.


What It Does

Sits between your applications and LLM providers, exposing a single OpenAI-compatible /v1 API.

  • Load distribution: Split traffic across multiple providers and provider accounts.
  • Access control: Issue separate API keys with individual model, provider, and rate limits.
  • Built-in dashboard: Full admin UI embedded into the go binary.

Architecture & Philosophy

  • Blazingly fast: Go + Svelte, no heavy JS runtime.
  • Modular: Core repo remains lightweight and maintainable. Providers and plugins live in separate repositories.
  • Everything works: Zero dead code, all features are tested by real people.
  • Clean UI: Fully reviewed Svelte dashboard polished by a human.
  • UX with common sense: No unmoderated AI slop that nobody knows how to use.

Key Features

Admin Panel
  • Chat: Test models directly in the browser.
  • Metrics: Inspect model and provider usage.
  • Providers: Connect new providers and accounts.
  • Models: Browse available models across active providers.
  • Agents: Configure multi-model routing rules visually.
  • Tokens: Manage access tokens with fine-grained model permissions.
Modular Builds

Include only the providers you need via adapters.conf:

# Example: github.com/TheSlopMachine/llm-router-adapter-demo

github.com/TheSlopMachine/llm-router-adapter-google
github.com/TheSlopMachine/llm-router-adapter-kiro
github.com/TheSlopMachine/llm-router-adapter-opencode-zen

Workspace setup is fully handled by the Makefile.

Zero-Config

No config files. Minimal set of optional CLI flags and you are done.

llm-router --help

llm-router — minimalist LLM routing gateway

Usage:
  llm-router [host] [flags]

Examples:
  llm-router
  llm-router localhost --web 8080 --api 8081 --db ./llm-router.db

Flags:
      --api Port             port for /v1 OpenAI-compatible API (default 8081)
      --db string            path to the database file (default "llm-router.db")
  -h, --help                 help for llm-router
      --log-level LogLevel   log level: debug, info, warn, error (default info)
      --testing-key string   path to file with bearer token (auto-generated)
  -v, --version              print version information and exit
      --web Port             port for dashboard UI (default 8080)

Quick Start

Development server
make start
make start HOST=localhost WEB_PORT=8080 API_PORT=8081
Production builds
make publish
make publish PUBLISH_PLATFORMS="windows/amd64 linux/amd64 darwin/amd64 freebsd/amd64"
Dashboard setup
  1. Create the initial admin account.
  2. Go to Providers and authenticate in any of listed providers, or add a new one.
  3. Go to Tokens and issue a new token. In wizard, select Allow all if you don't care about limits.
  4. The token will be shown to you once, afterwards you won't be able see it again.
API usage

Pass your token and the endpoint URL to your harness or any other AI-powered software:

  • Base URL: http://localhost:8081/v1
  • Token: 10b7****************************************************4fd7

License

MIT

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Package cmd defines the llm-router CLI using cobra.
Package cmd defines the llm-router CLI using cobra.
internal
adapters/generic
Package generic provides a generic OpenAI-compatible backend for custom providers.
Package generic provides a generic OpenAI-compatible backend for custom providers.
api/v1
Package v1 implements the OpenAI-compatible /v1/...
Package v1 implements the OpenAI-compatible /v1/...
config
Package config holds the runtime configuration populated from CLI flags.
Package config holds the runtime configuration populated from CLI flags.
dashboard
Package dashboard serves the admin SPA and its backing JSON API.
Package dashboard serves the admin SPA and its backing JSON API.
db
Package db manages the embedded bbolt database and all bucket definitions.
Package db manages the embedded bbolt database and all bucket definitions.
errors
Package errors defines standard error types used across llm-router services.
Package errors defines standard error types used across llm-router services.
models
Package models defines all shared data structures for llm-router.
Package models defines all shared data structures for llm-router.
repository
Package repository provides generic type-safe database operations using Go generics.
Package repository provides generic type-safe database operations using Go generics.
server
Package server wires all services together and starts the HTTP servers.
Package server wires all services together and starts the HTTP servers.
services/admin
Package admin manages the bootstrap flow and dashboard authentication.
Package admin manages the bootstrap flow and dashboard authentication.
services/config
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
Package config manages RouterConfiguration persisted in the RouterConfiguration bucket.
services/credential
Package credential implements the Credential Pool Service.
Package credential implements the Credential Pool Service.
services/geoip
Package geoip resolves the server's country for proxy preference matching.
Package geoip resolves the server's country for proxy preference matching.
services/maintenance
Package maintenance implements the Provider Maintenance & Rotation Service.
Package maintenance implements the Provider Maintenance & Rotation Service.
services/metrics
Package metrics implements the Metrics Service for tracking API usage.
Package metrics implements the Metrics Service for tracking API usage.
services/modelinfo
Package modelinfo manages model metadata caching.
Package modelinfo manages model metadata caching.
services/pluginrepo
Package pluginrepo manages external plugin repositories (the plugin store).
Package pluginrepo manages external plugin repositories (the plugin store).
services/provider
Package provider owns the unified provider registry.
Package provider owns the unified provider registry.
services/proxypool
Package proxypool implements the proxy pool service.
Package proxypool implements the proxy pool service.
services/retry
Package retry implements the single retry/fallthrough engine shared by the router (credential rotation) and the agents adapter (model rotation).
Package retry implements the single retry/fallthrough engine shared by the router (credential rotation) and the agents adapter (model rotation).
services/router
Package router implements the Router Service.
Package router implements the Router Service.
services/token
Package token implements the Internal Token Service.
Package token implements the Internal Token Service.
services/virtual
Package virtual implements the Virtual Models service.
Package virtual implements the Virtual Models service.
testutil
Package testutil provides shared test utilities for llm-router tests.
Package testutil provides shared test utilities for llm-router tests.
util
Package util provides shared utility functions for the llm-router framework.
Package util provides shared utility functions for the llm-router framework.
providers
virtual
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.
Package virtual implements the virtual-model provider backend: a virtual model routes each request to its models in list order (fall-through) and fails only when every model failed.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL