yokai

README

yokai

One binary to deploy, monitor, and manage LLM inference across all your GPUs.

yokai is a terminal-based fleet manager for running vLLM, llama.cpp, and ComfyUI on any number of GPU machines. Connect your devices, deploy curated Best-Known-Configs (or roll your own through a guided wizard), and watch everything on a btop-style dashboard -- all from a single binary with zero dependencies on the target machines.

 ┌───────────┐ ╔══════════════════╗ ┌──────────────┐ ┌─────────────┐ ┌───────────────┐
 │  G. Home  │ ║  ▸ 1. Dashboard  ║ │  2. Devices  │ │  3. Deploy  │ │  4. Settings  │
 └───────────┘ ╚══════════════════╝ └──────────────┘ └─────────────┘ └───────────────┘

 ╭────────────────────────────────────────────────────────────────────────────────────╮
 │ Dashboard                                                                          │
 │ Fleet services, inspector details, and contextual live logs.                       │
 │                                                                                    │
 │ ┌─ AI Fleet ───────────────────────────────────────────────────────────────────┐   │
 │ │ GPU  73%  ▁▂▃▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆▅▆▇█▇▆ │   │
 │ │ VRAM 84%  20.1 / 24.0 GB                                                     │   │
 │ │ GPUs 2 active / 2 total · Svc 3 total · 0 alert(s)                           │   │
 │ └──────────────────────────────────────────────────────────────────────────────┘   │
 │                                                                                    │
 │ ╔═ AI Services [2] ════════════════════════════════════════════════════════════╗   │
 │ ║ ▌ vllm-llama-3.1-8b      · Gaming Rig   · live                               ║   │
 │ ║ ●  llamacpp-mistral-7b   · Gaming Rig   · live                               ║   │
 │ ╚══════════════════════════════════════════════════════════════════════════════╝   │
 │                                                                                    │
 │ ┌─ Monitoring Services [1] ────────────────────────────────────────────────────┐   │
 │ │ ●  grafana                · Gaming Rig   · live                              │   │
 │ └──────────────────────────────────────────────────────────────────────────────┘   │
 ╰────────────────────────────────────────────────────────────────────────────────────╯

 ╭────────────────────────────────────────────────────────────────────────────────────╮
 │ G home  |  Tab next section  |  Shift+Tab previous section  |  1-4 jump            │
 ╰────────────────────────────────────────────────────────────────────────────────────╯

---

Why yokai?

If you're running local LLMs across multiple machines, you already know the pain:

• SSH into each box just to check if your GPU is melting or idle
• Copy-paste 20-flag docker run commands every time you want to swap a model
• Juggle separate monitoring dashboards for each machine
• Manually edit tool configs every time an endpoint changes

yokai solves all of this with a single binary. Install it, point it at your machines, and you're running models in minutes -- not hours.

---

Features

Fleet Management

• Onboarding wizard -- connect devices via LAN scan, Tailscale peer discovery, or manual IP entry
• SSH bootstrap -- pre-flight checks (Docker, GPU, disk space), agent deployment, and systemd service installation in one step
• Device manager -- add, edit, remove, and test connectivity for all devices from the TUI
• Secure by default -- auto-generated bearer tokens for agent authentication, SSH key resolution with agent/key/password fallback

Workload Deployment

• Best Known Configs (BKC) -- pick from a built-in catalog of pre-validated vLLM and llama.cpp deploys grouped by vendor (NVIDIA, OpenAI, Meta/Llama, Google, Mistral, Qwen, DeepSeek, GLM, Moonshot, Microsoft, and more) and filtered to the GPUs each device actually has
• Guided deploy wizard -- pick workload type, target device, Docker image, model, and runtime config when you want to deviate from the catalog
• HuggingFace integration -- search models directly, browse GGUF quantizations, auto-download during deployment
• VRAM estimator -- hf-mem-backed memory estimate for vLLM weights + KV cache before you commit to a deploy
• Docker image catalog -- browse official tags from Docker Hub and GHCR, including nightly builds
• Plugin system -- model-specific add-ons (e.g. Nemotron Super V3 reasoning parser) that fetch assets, mount them into the container, and append the right runtime flags
• GPU-aware deployment -- automatic --gpus flag configuration, multi-GPU tensor parallelism support

Live Monitoring

• btop-style dashboard -- live GPU utilization, VRAM, temperature, power draw, and fan speed per GPU
• System metrics -- CPU/RAM sparklines with 60-sample rolling history, disk usage, swap
• Container metrics -- per-container CPU, memory, GPU memory, and uptime
• Service lifecycle -- stop, restart, and remove containers directly from the dashboard

Monitoring Stack

• Auto-provisioned -- Prometheus + Grafana + node_exporter + dcgm-exporter seeded onto each device during bootstrap, with the agent bearer token wired in for authenticated scrapes
• Pre-built dashboards -- Grafana dashboard with GPU utilization, temperature, power, and system metrics panels
• Live in the dashboard -- the monitoring stack shows up as its own services panel alongside your AI workloads

AI Coding Tool Integration

• One-shot config -- the Settings route auto-configures every supported AI coding tool at once
• VS Code Copilot -- appends OpenAI-compatible models to chat.models[] in the VS Code user settings.json
• OpenCode -- registers a per-host provider in ~/.config/opencode/opencode.json via @ai-sdk/openai-compatible
• OpenClaw -- adds a yokai provider under models.providers in ~/.openclaw/openclaw.json
• Claude Code -- writes ANTHROPIC_BASE_URL, ANTHROPIC_MODEL, and ANTHROPIC_CUSTOM_MODEL_OPTION into ~/.claude/settings.json
• Codex -- adds a [model_providers.yokai] section to ~/.codex/config.toml
• Backup-safe -- writes a .yokai.bak of every config before modifying it
• Multi-endpoint -- registers every running inference service as an available endpoint

Self-Updating

• yokai upgrade -- checks GitHub Releases, downloads the correct binary for your OS/arch, and replaces itself in place
• Cross-platform -- builds for Linux, macOS, and Windows (amd64 + arm64)
• One-line install -- curl | sh installer that detects your platform automatically

---

Quick Start

Install

One-line installer (Linux/macOS):

curl -fsSL https://raw.githubusercontent.com/spencerbull/Yokai/main/install.sh | sh

From GitHub Releases:

Download the latest binary for your platform from the Releases page, extract it, and move it to your PATH.

Build from source:

git clone https://github.com/spencerbull/yokai.git
cd yokai
make build
./bin/yokai

First Run

# 1. Launch yokai -- OpenTUI starts and auto-starts the local daemon if needed
yokai

# 2. Select how to connect (LAN / Tailscale / Manual IP)
# 3. Enter SSH credentials -- yokai tests the connection
# 4. Bootstrap runs: pre-flight checks → agent deploy → monitoring stack
# 5. Set up your HuggingFace token (optional, for gated models)
# 6. Tab over to "Deploy" and pick a Best-Known-Config to run your first model

Running the Daemon

The daemon is a background process on your local machine that maintains SSH tunnels to your devices and polls agents for live metrics. yokai now auto-starts it on first launch, but you can still run it manually:

# Start the daemon in the background
yokai daemon &

# Launch OpenTUI (auto-starts daemon if needed)
yokai

---

Usage

Commands

Core

Command	Description
yokai	Launch OpenTUI (default; auto-starts the daemon)
yokai agent [port]	Run the agent on a target device (default port: 7474)
yokai daemon	Start the local background daemon (default 127.0.0.1:7473)
yokai upgrade	Update to the latest release
yokai version	Print version and build info

Device management

Command	Description
yokai devices list	List all configured devices
yokai devices add --host <host> [flags]	Add a device
yokai devices remove <device-id>	Remove a device
yokai devices test <device-id>	Test SSH + agent connectivity
yokai devices bootstrap <device-id>	Install or upgrade the agent on a device

Service management

Command	Description
yokai services list [--device <id>]	List containers across the fleet (or one device)
yokai services deploy --device <id> [flags]	Deploy a service
yokai services stop <device-id> <cid>	Stop a container
yokai services restart <device-id> <cid>	Restart a container
yokai services logs [--follow] <did> <cid>	Stream container logs

Fleet status & config

Command	Description
yokai status	Fleet overview (JSON)
yokai metrics [--device <id>]	Detailed device metrics (JSON)
yokai config show	Dump config (tokens redacted)
yokai config set <key> <value>	Set a config value
yokai config path	Print the config file path

All non-TUI commands emit JSON on stdout and JSON-formatted errors on stderr, so they're scriptable from shell or another tool.

Keybinds

Global

Key	Action
Tab / Shift+Tab	Cycle between Dashboard / Devices / Deploy / Settings
1 … 4	Jump to a top-level route by number
g	Return to the landing screen
Esc (twice) or Ctrl+C (twice)	Quit

Dashboard

Key	Action
j / k (or ↑ / ↓)	Move selection within the service list
Tab	Switch between AI services and the monitoring stack
Enter or l	Open the selected service's detail view
Shift+L	Open logs for the selected service
s / r / t / x	Stop / restart / test / delete (in service detail)
y / n	Confirm or cancel a destructive action

---

How It Works

yokai uses a three-tier architecture: OpenTUI (what you see, a TypeScript app rendered in your terminal), the Daemon (runs locally and brokers everything), and the Agent (runs on each GPU device).

Your Machine                              GPU Device(s)
┌──────────────────────┐                  ┌──────────────────────┐
│  OpenTUI (Bun/Node)  │                  │  yokai agent         │
│  ├── Dashboard       │   HTTP :7473     │  ├── REST API :7474  │
│  ├── Deploy Wizard   │◄───────────────► │  ├── nvidia-smi      │
│  ├── Device Manager  │                  │  ├── Docker engine   │
│  └── Log Viewer      │                  │  └── System metrics  │
│         ▲            │                  │                      │
│         │ launches   │                  │  Docker containers   │
│         ▼            │   SSH tunnel     │  ├── vLLM :8000      │
│  yokai daemon        │◄═══════════════► │  ├── llama.cpp :8080 │
│  ├── SSH Tunnels     │                  │  └── ComfyUI :8188   │
│  ├── Metrics Agg.    │                  │                      │
│  ├── BKC Catalog     │                  │  Monitoring stack    │
│  ├── HF + hf-mem     │                  │  ├── Prometheus      │
│  └── Tool Configs    │                  │  ├── Grafana         │
│                      │                  │  ├── node_exporter   │
│  ~/.config/yokai/    │                  │  └── dcgm-exporter   │
│  └── config.json     │                  │                      │
└──────────────────────┘                  └──────────────────────┘

Launch sequence

1. yokai checks whether the daemon is healthy; if not, it spawns yokai daemon in the background and waits for /health
2. It then runs the bundled yokai-tui binary (preferred) or, if you're on a dev checkout with bun installed, bun run src/index.tsx from ui/tui/ -- the daemon URL is passed via YOKAI_DAEMON_URL
3. The TUI talks to the daemon over HTTP on 127.0.0.1:7473

Data flow

1. The daemon opens SSH tunnels to each device and forwards a local port to each agent's :7474 REST API
2. Every few seconds it polls each agent's /metrics endpoint and caches the results in memory
3. The TUI reads aggregated metrics from the daemon and renders the dashboard
4. Deploy and lifecycle commands (deploy/stop/restart/remove/logs) flow TUI -> daemon -> agent -> Docker

---

Configuration

All state lives in ~/.config/yokai/config.json. Copy this file to another machine to reconnect to your fleet instantly.

{
  "version": 1,
  "hf_token": "hf_...",
  "daemon": {
    "listen": "127.0.0.1:7473",
    "metrics_poll_interval_s": 2,
    "reconnect_interval_s": 30
  },
  "devices": [
    {
      "id": "gaming-rig",
      "label": "Gaming Rig",
      "host": "100.64.0.2",
      "ssh_user": "user",
      "connection_type": "tailscale",
      "agent_port": 7474,
      "agent_token": "a1b2c3...",
      "gpu_type": "nvidia",
      "tags": ["rtx-4090"],
      "monitoring_installed": true
    }
  ],
  "services": [
    {
      "id": "yokai-vllm-llama3",
      "device_id": "gaming-rig",
      "type": "vllm",
      "image": "vllm/vllm-openai:latest",
      "model": "meta-llama/Llama-3.1-8B-Instruct",
      "port": 8000,
      "plugins": [],
      "runtime": { "ipc_mode": "host", "shm_size": "16g" }
    }
  ],
  "preferences": {
    "theme": "tokyonight",
    "default_vllm_image": "vllm/vllm-openai:latest",
    "default_llama_image": "ghcr.io/ggml-org/llama.cpp:server-cuda",
    "default_comfyui_image": "yanwk/comfyui-boot:latest"
  }
}

---

Best Known Configurations (BKC)

The BKC catalog is a library of pre-validated deploy recipes. Each entry pins the Docker image, tensor-parallel size, quantization flags, GPU memory utilization, chat template, tool-call parser, and any runtime options (--ipc=host, --shm-size, ulimits) needed for a given model on a given GPU. The deploy wizard matches your model against the catalog, filters recipes by the target device's VRAM and GPU count, and offers one-click apply.

86 vLLM configs across 82 models from 23 publishers (as of this commit). Entries are grouped by publisher and live in internal/bkc/catalog_*.go:

Publisher	Unique models	vLLM configs	Catalog file
Qwen	20	20	catalog_qwen.go
nvidia	12	14	catalog_nvidia.go, catalog_llama.go, catalog_moonshotai.go, catalog_qwen.go
zai-org (GLM)	7	7	catalog_glm.go
deepseek-ai	6	6	catalog_deepseek.go
google	5	5	catalog_google.go
mistralai	5	5	catalog_mistral.go
baidu (ERNIE)	4	4	catalog_others.go
moonshotai	4	4	catalog_moonshotai.go
openai	2	4	catalog_openai.go
meta-llama	2	2	catalog_llama.go
internlm	2	2	catalog_others.go
tencent	2	2	catalog_others.go
microsoft (Phi)	1	1	catalog_microsoft.go
amd	1	1	catalog_openai.go
ByteDance-Seed	1	1	catalog_others.go
MiniMaxAI	1	1	catalog_others.go
OpenGVLab (InternVL)	1	1	catalog_others.go
PaddlePaddle	1	1	catalog_others.go
XiaomiMiMo	1	1	catalog_others.go
arcee-ai	1	1	catalog_others.go
inclusionAI	1	1	catalog_others.go
jinaai	1	1	catalog_others.go
stepfun-ai	1	1	catalog_others.go
Total	82	86

Notable recipes

• Frontier reasoning / MoE — DeepSeek R1 / V3.1 / V3.2 (FP8, 8× H200/B200), Qwen3-Coder-480B-A35B, Qwen3.5-397B-A17B-FP8, Qwen3-Next-80B-A3B (BF16 and FP8 variants), Moonshot Kimi K2.
• Gated Meta / NVIDIA — Llama 3.3 70B (BF16 on Meta base, FP8 on Hopper, FP4 on Blackwell), nvidia/Llama-3.1-Nemotron-Ultra / Super, NGC images validated on DGX Spark GB10 and Jetson Thor (aarch64).
• Vision-language — Qwen2.5-VL-7B/72B, Qwen3-VL-235B (BF16 + FP8), InternVL3.5-8B, ERNIE-4.5-VL-28B / VL-424B, PaddleOCR-VL, DeepSeek-OCR.
• Small / edge-friendly — Qwen3-0.6B / 1.7B / 4B / 8B, Qwen3Guard-Gen-0.6B, Phi-4, Gemma 3 2B/4B/12B, validated for RTX 4090, RTX 5090, L40S, GB10, and Jetson Thor.
• AMD CDNA4 — amd/gpt-oss-120b-w-mxfp4-a-fp8 tuned for MI355X with the ROCm vLLM image.

Device-aware selection

BKC entries carry explicit hardware tags so the deploy wizard can hand you the right recipe for the device you selected:

• NVIDIA: gb10 (DGX Spark), jetson-thor, rtx-pro-6000, rtx-5090, rtx-4090, l40s, a100-80, h100-80, h100-94, h200, h20, b200, gb200
• AMD: mi300x, mi325x, mi355x, radeon-r9700

When multiple BKCs target the same model (e.g. Llama 3.3 70B has BF16, FP8, and FP4 variants), the daemon prefers the most specialised recipe whose TargetDevices include your device profile, then falls back to the first recipe whose MinVRAMGBPerGPU and MinGPUCount the device can satisfy.

Adding a BKC

Drop a new entry into the appropriate catalog_<vendor>.go file (create one if your vendor doesn't have one yet) and register it in the catalog slice in catalog_data.go. Every entry needs a stable ID, a human-readable Name, the Docker image, port, exact runtime flags, and the hardware gates (TargetDevices, MinVRAMGBPerGPU, MinGPUCount, Quantization, Arch). See catalog_llama.go for a multi-variant example.

---

Supported Platforms

User Machine (where you run yokai)

OS	Architecture	Status
Linux	amd64	Supported
Linux	arm64	Supported
macOS	amd64	Supported
macOS	arm64 (Apple Silicon)	Supported
Windows	amd64	Supported
Windows	arm64	Supported

Target Devices (where models run)

Requirement	Details
OS	Linux (Ubuntu 20.04+, Debian 11+, or similar)
Docker	20.10+ with the Docker CLI available
GPU	NVIDIA with nvidia-container-toolkit installed
Network	Reachable via SSH (LAN, Tailscale, or public IP)
Disk	20GB+ free recommended (vLLM images are 10GB+)

---

Architecture Documentation

Detailed multi-level architecture docs are available in the architecture/ directory:

Level	Document	What It Covers
L1	System Overview	Bird's-eye view of the system
L2	Component Architecture	Go package structure and dependencies
L3	Data Flow	Metrics, deployment, logging, and config data paths
L4	Network Topology	SSH tunnels, ports, authentication, Tailscale
L5	TUI Screen Map	View hierarchy and navigation state machine
L6	Agent API	Full REST API contract with JSON schemas
L7	Daemon UI API	Daemon REST API consumed by OpenTUI

---

Project Structure

yokai/
├── cmd/yokai/             # Binary entry point and subcommand dispatch
├── internal/
│   ├── agent/             # Remote agent: REST API, Docker ops, system metrics (port :7474)
│   ├── bkc/               # Best Known Configs catalog (per-vendor catalog_*.go files)
│   ├── claudecode/        # Claude Code (~/.claude/settings.json) endpoint registration
│   ├── cli/               # Non-TUI subcommand handlers (devices/services/status/metrics/config)
│   ├── codex/             # Codex (~/.codex/config.toml) endpoint registration
│   ├── config/            # Config load/save/migrate, deploy history (~/.config/yokai/)
│   ├── daemon/            # Local daemon: REST API (:7473), SSH tunnels, metrics aggregation
│   ├── docker/            # Docker Hub/GHCR tag catalog and image helpers
│   ├── hf/                # HuggingFace API: model search, GGUF listing
│   ├── hfmem/             # `hf-mem` wrapper for vLLM weight + KV-cache memory estimation
│   ├── monitoring/        # Provisions Prometheus/Grafana/exporters onto remote devices
│   ├── openclaw/          # OpenClaw openclaw.json provider registration
│   ├── opencode/          # OpenCode opencode.json provider registration
│   ├── opentui/           # OpenTUI launcher (daemon health check + bundled/Bun spawn)
│   ├── platform/          # Cross-platform shims (e.g. chmod no-op on Windows)
│   ├── plugins/           # Plugin catalog: assets, mounts, runtime-flag overrides per model
│   ├── ssh/               # SSH client, SCP upload, bootstrap/deploy
│   ├── tailscale/         # Tailscale CLI wrapper for peer discovery
│   ├── upgrade/           # Self-update from GitHub Releases
│   └── vscode/            # VS Code settings.json (chat.models[]) manipulation
├── ui/tui/                # OpenTUI: TypeScript/React TUI run by Bun
│   ├── src/
│   │   ├── app/           # App shell and routing
│   │   ├── contracts/     # Daemon API types
│   │   ├── features/      # Dashboard, deploy, devices, logs, integrations, etc.
│   │   ├── services/      # Daemon HTTP client
│   │   └── theme/         # Color palette and styles
│   └── package.json
├── assets/
│   ├── grafana/           # Pre-built dashboard JSON and provisioning
│   ├── prometheus/        # Prometheus scrape configuration
│   └── systemd/           # Agent systemd service template
├── architecture/          # Multi-level architecture documentation
├── docker/                # Reference Dockerfiles (e.g. comfyui/)
├── .github/workflows/     # CI (build/test/lint) and release (GoReleaser)
├── .goreleaser.yml        # Cross-compilation and release config
├── install.sh             # curl-pipe-sh installer
├── Makefile               # Build, test, lint, cross-compile targets
└── go.mod

---

Contributing

Contributions are welcome. See CONTRIBUTING.md for development setup, coding guidelines, and the pull request process.

---

License

MIT