hydrallm

README

HydraLLM

English | 简体中文 | 日本語

flowchart LR
  C["Retry Current Model"] -->|"fail"| D["Fallback Next Model"];
  D -->|"next"| C;
  C -->|"success"| E["✅ Return Response"];
  D -->|"exhausted"| F["❌ Return Error"];

  classDef muted fill:#F8FAFC,stroke:#CBD5E1,color:#64748B,stroke-width:1px;
  classDef flow fill:#FFF7ED,stroke:#EA580C,color:#7C2D12,stroke-width:3px;
  classDef flow2 fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:3px;
  classDef ok fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:1.5px;
  classDef bad fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:1.5px;
  class C flow;
  class D flow2;
  class E ok;
  class F bad;

HydraLLM is a high-performance LLM API proxy with automatic retry and model fallback across OpenAI-compatible, Anthropic, and AWS Bedrock providers.

When a request fails, HydraLLM retries the current model, then falls back to the next configured model until success or exhaustion.

[!TIP] For complete configuration fields and examples, see CONFIGURATION.md.

✨ Why HydraLLM

• Automatic retry + fallback for coding and agent workloads.
• Multi-provider support: OpenAI-compatible, Anthropic, and AWS Bedrock.
• Single local endpoint with stable client integration while model chains evolve.

📦 Install

Homebrew (macOS / Linux)

brew install fang2hou/tap/hydrallm

Install via Go (All platforms)

go install github.com/fang2hou/hydrallm@latest

Binary Download (All platforms)

Download from GitHub Releases.

🚀 Quick Start (GLM Coding Plan)

The GLM showcase included with this project exposes two listeners in one config:

• OpenAI-compatible API on http://127.0.0.1:8101
• Anthropic-compatible API on http://127.0.0.1:8102

1) Prepare config

macOS / Linux:

mkdir -p ~/.config/hydrallm
curl -o ~/.config/hydrallm/config.toml \
  https://raw.githubusercontent.com/fang2hou/hydrallm/main/showcases/glm-coding-plan.toml

Windows (PowerShell):

New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\hydrallm"
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/fang2hou/hydrallm/main/showcases/glm-coding-plan.toml" -OutFile "$env:USERPROFILE\.config\hydrallm\config.toml"

2) Set API key

macOS / Linux:

export ZAI_API_KEY="your-api-key"

Windows (PowerShell):

$env:ZAI_API_KEY = "your-api-key"

3) Start proxy

hydrallm

4) Verify listeners

OpenAI-compatible listener (8101)

curl http://127.0.0.1:8101/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "placeholder",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

Anthropic-compatible listener (8102)

curl http://127.0.0.1:8102/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "placeholder",
    "max_tokens": 64,
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

[!NOTE] HydraLLM overrides request model with the configured model chain for each listener.

🔁 Service (Auto Start on Boot)

Use Homebrew services for auto-start.

[!NOTE] For brew services, configure api_key explicitly in the config file; do not rely on shell environment variables.

brew services start hydrallm
brew services info hydrallm
brew services restart hydrallm
brew services stop hydrallm

• macOS: launchd (starts automatically after login)
• Linux: systemd

🛠️ CLI Commands

Command	Description
hydrallm	Start server
hydrallm serve	Start proxy
hydrallm edit	Open config in $EDITOR
hydrallm version	Print version info
hydrallm --help	Show help

Global flags: --config /path/to/config.toml, --log-level info

🧯 Troubleshooting

Quick diagnostics:

hydrallm --config /path/to/config.toml --log-level debug
brew services list | grep hydrallm

config validation failed: at least one model must be configured

Add at least one model under [models.<id>].

model "...": provider "..." not found

Ensure each model provider matches a key under [providers.<name>].

listener "...": mixed model types are not allowed

Each listener must contain models of a single API type (openai, anthropic, or bedrock). Split mixed types across multiple listeners.

Requests return upstream 4xx/5xx

Temporarily set log.include_error_body = true to inspect upstream error details.

📄 License

MIT