JanusLLM

README

JanusLLM

JanusLLM is an AI gateway designed to streamline interactions with multiple Large Language Model (LLM) APIs through one unified entry point.

Architecture

Features

JanusLLM offers a robust set of features to enhance the efficiency, scalability, and cost-effectiveness of LLM usage:

• Unified API Gateway for OpenAI/Anthropic LLM providers
• Advanced Load Balancing: Supports round-robin and weighted policies.
• Billing and Cost Management: Tracks token usage and spend by request/key/team/org/model, and updates key balance/total spend.
• Usage Limits and Quotas: Supports API key auth, model permission checks, key expiration, balance checks, and per-key RPM rate limiting.

Frontend Admin UI

The admin dashboard lives in web/ and is built with Vite, React, and TypeScript. It currently uses mock dashboard data, with API client helpers prepared for /v1/admin resources and /v1/models.

cd web
npm install
npm run dev

By default, the Vite dev server proxies API calls to http://localhost:8080. Override it when needed:

$env:VITE_JANUS_API_BASE_URL="http://localhost:8080"
npm run dev

On Unix-like shells, the one-line form also works:

VITE_JANUS_API_BASE_URL=http://localhost:8080 npm run dev

Client
  |
  v
Gin Gateway
  |-- Auth middleware
  |-- Spend middleware
  |-- Proxy adapters
  |-- Balancer
  v
Provider endpoints

PostgreSQL stores auth, admin, model metadata, and billing records.

Current Status

JanusLLM is currently an MVP gateway. Runtime routing is loaded from config/config.yaml; PostgreSQL is used for auth, admin, billing, and auxiliary model metadata.

This branch aligns docs with code and adds:

• startup synchronization from YAML model config into database model tables,
• graceful handling for old auth helpers that previously used log.Fatal,
• a modern admin frontend,
• an extensible balancer interface with latency-based and client-sticky strategies,
• richer spend log fields for provider, latency, cache hit, and tenant.

Quickstart

1. Install prerequisites:

• Go 1.24.3 or newer
• PostgreSQL 14 or newer

1. Initialize the database:

psql -h <PG_HOST> -p <PG_PORT> -U <PG_USER> -d <DB_NAME> -f scripts/db/create_core_tables.sql

1. Configure the service:

cp config/config.yaml.example config/config.yaml

Fill in:

• service.port
• secrets.database_url
• admin.master_key
• models.model_groups

Environment variables override local secrets when present:

• JANUS_DATABASE_URL
• JANUS_ADMIN_MASTER_KEY

1. Run the gateway:

go mod tidy
go run ./cmd

1. Test a proxied request:

curl --location 'http://127.0.0.1:8080/v1/chat/completions' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <your_api_key>' \
  --data '{
    "model": "<model_group_name>",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": false,
    "temperature": 0.7,
    "max_tokens": 4096
  }'

Admin API

On startup, JanusLLM creates or updates the built-in admin user admin using admin.master_key or JANUS_ADMIN_MASTER_KEY.

curl -u admin:<ADMIN_MASTER_KEY> http://127.0.0.1:8080/v1/admin/organizations
curl http://127.0.0.1:8080/swagger/openapi.json

Swagger UI is available at:

http://127.0.0.1:8080/swagger/

Configuration Priority

• config/config.yaml is the source of truth for runtime model routing.
• PostgreSQL model tables are an auxiliary management and audit view.
• Startup synchronization should upsert YAML model groups/endpoints into PostgreSQL and disable records that no longer exist in YAML.
• Upstream API keys should stay in config/secret storage; database rows should store secret references or empty values, not plaintext provider keys.