llama-launcher

README

llama-launcher

A terminal tool for managing local LLM servers through named configuration profiles. Supports llama.cpp, Ollama, and LM Studio as backends. Define your models and parameters once in a YAML file, then load and switch between them with a single command or an interactive TUI.

llama-launcher is a process manager, not a request router: it starts and stops LLM servers and tells them which model to load. Clients talk to each server directly via its native address. The launcher exits after dispatching work, consuming zero resident memory while the server runs.

Install

Homebrew (macOS)

brew tap airiclenz/tap
brew install llama-launcher

This installs the llama-launcher CLI and the optional llama-launcher-mcp control-plane adapter (see Remote control from a container). The adapter is inert until you start it.

From source

Requires Go 1.26+.

make build      # => ./llama-launcher, for local testing

Installation is deliberately Homebrew-only; make install just points you there.

Quick start

# First run generates an example config
llama-launcher
# => Created example config at: ~/.config/llama-launcher/config.yaml

# Edit the config with your model paths, then run again
llama-launcher

Configuration

The config lives at ~/.config/llama-launcher/config.yaml (override with --config or LLAMA_LAUNCHER_CONFIG). The generated example is fully commented and documents every option — internal/launcher/defaults/config.yaml is the complete reference. A minimal config looks like this:

servers:
  llamacpp: true

models_dir: ~/Models     # base directory for model files (llamacpp)

defaults:                # shared by all profiles
  gpu_layers: 99
  threads: 8
  context_size: 8192

profiles:
  qwen-coder:
    title: "Qwen 2.5 Coder 32B"
    server: llamacpp
    model: qwen2.5-coder-32b-q4_k_m.gguf   # relative to models_dir
    context_size: 32768
    is_favourite: true                     # pinned to the top of menus

  llama-8b:
    server: llamacpp
    model: llama-3.1-8b-instruct-q5_k_m.gguf

Parameters merge in three tiers: profile > defaults > built-in fallbacks. "Not set" is always distinct from zero. Not every parameter applies to every backend — the commented example config has the full parameter/backend matrix.

Other top-level options control launcher behaviour (auto_stop_server, auto_unload, log_retention) and the TUI (display_centered, auto_close, sort_alphabetically, refresh_duration, and the memory readout below).

Backends

Backend	Default address	Model reference
llamacpp	127.0.0.1:8080	File path (relative to models_dir or absolute)
ollama	localhost:11434	Ollama model name (e.g. llama3.1:8b, pulled first)
lmstudio	localhost:1234	LM Studio model key (e.g. lmstudio-community/meta-llama-3.1-8b-instruct)

For each backend, the launcher knows how to start the server (fork-and-detach for llamacpp; ollama serve for Ollama; lms server start for LM Studio) and how to stop it. stop is unconditional — the launcher does not distinguish servers it started from servers that were already running (see ADR-0001). Multiple instances may run concurrently as long as each binds a distinct host:port.

API keys

Each entry in the servers section can carry an optional API key by switching from the bool form to the mapping form (enabled defaults to true when omitted):

servers:
  llamacpp:
    api_key: "secret"
  lmstudio:
    enabled: true
    api_key: "lm-studio-abc123..."
  ollama: false

The key doesn't have to live in the file. api_key_cmd names a command whose standard output is the key — typically a lookup in your secret store:

servers:
  llamacpp:
    api_key_cmd: "security find-generic-password -s llama-launcher -a llamacpp -w"

The command is handed whole to a shell (pipelines work) and runs once at startup for every enabled server. Set api_key or api_key_cmd, never both. If you do want a literal key to stay in the file, set plaintext_key_ok: true on the entry so nothing offers to move it into a secret store.

While a literal key sits in the file without plaintext_key_ok: true, running llama-launcher with no arguments raises one offer before the menu — move the key(s) into your machine's secret store (the entry's api_key line becomes an api_key_cmd line, and only after the stored key has been read back through it), not now (asked again next launch), or never for these entries (records plaintext_key_ok: true). Subcommands never prompt, so they print a one-line warning naming the entries, the config file this run read, and the ways out by hand instead.

The launcher is not a proxy, so what the key does depends on the backend:

Backend	Effect of api_key
llamacpp	Exported as LLAMA_API_KEY into the launched server's environment — llama-server then rejects client requests without Authorization: Bearer <key> (its /health endpoint stays open). Deliberately never put on the command line, so the key does not show up in ps.
lmstudio	LM Studio manages its own token: enable Require API token in its Server Settings, generate a token there, and paste it here so the launcher's health checks and model loads keep working.
ollama	Ollama has no native authentication. Set a key only when the instance sits behind an authenticating reverse proxy; the launcher then sends it with its own requests.

In all cases the launcher attaches the key as a Bearer header to the HTTP calls it makes itself (health checks, model load/unload, model listing). The config file is created with mode 0600. For llamacpp, llama-server reads LLAMA_API_KEY only when no --api-key flag is given, so an extra_args --api-key override still wins — but that literal override is visible in ps.

Memory readout

The TUI's status header shows a live memory + swap readout (macOS), fully customizable via memory_status_format — colored spans, 24-bit colors, and bar-graph gauges:

memory_status_format: "{bold}Free RAM:{reset} {yellow}{free_ram} {bright-blue}{free_ram_pct}{reset} {used_ram_pct:bar} ✦ {bold}Swap:{reset} {yellow}{swap_used}{reset} ✦ {bold}GPU:{reset} {gpu_util_pct:bar}"

Placeholders, style tags, and bar syntax

Placeholders

Placeholder	Value
{free_ram}	Available RAM (free + inactive + speculative + purgeable pages), humanised
{used_ram}	total_ram - free_ram, humanised
{total_ram}	Total physical RAM, humanised
{compressed_ram}	Bytes held by the kernel's memory compressor, humanised
{swap_used} / {swap_total} / {free_swap}	Swap in use / allocated / remaining, humanised
{free_ram_pct} / {used_ram_pct}	Rounded integer percentages of total RAM (e.g. 38%)
{swap_used_pct}	Percentage of allocated swap; 0% when swap is disabled
{gpu_util_pct}	GPU Device Utilization % from ioreg (Apple Silicon only)
{gpu_used_ram} / {gpu_alloc_ram}	Unified RAM held by / allocated to the GPU (Apple Silicon only)

Byte values are rendered macOS-style: 1024-based units with one decimal (12.4GB), whole values drop the decimal (8GB). Unknown placeholders are left in place.

Style tags

Tags	Effect
{black} {red} {green} {yellow} {blue} {magenta} {cyan} {white} {gray}	Standard ANSI colors
{bright-red} … {bright-white}	Bright ANSI variants
{0} … {255}	256-color palette index, e.g. {208}
{#rrggbb}	Exact 24-bit color, e.g. {#7aa2f7} (short {#rgb} works too)
{bold} {dim} {reset}	Text styles / back to terminal default

Named colors are resolved by your terminal theme; palette-index and hex colors render the same everywhere. A template without style tags or bars keeps the classic all-dim rendering; as soon as it contains one, you control all styling yourself.

Bar graphs

Any percentage placeholder can render as a value-less bar gauge: {pct_name:bar[:width[:color[:bgcolor]]]}. Trailing parts are optional and fall back to the memory_status_bar block:

memory_status_bar:    # defaults for every {..._pct:bar} token
  width: 10           # cells, clamped to 1–40
  color: green        # filled portion
  background: gray    # empty portion

Bars fill with block glyphs (eighth-block partials give 8 fill levels per cell) against a solid background — one continuous strip. Colors accept the same three forms as style tags. Malformed tokens are passed through literally, so typos are visible rather than silently dropped.

See the technical design doc for full schema details and behavior.

Usage

Interactive mode

Run without arguments to get the TUI menu. It adapts to three states:

• Stopped — select a profile to start the server and load a model
• Running with model — switch models, unload model, stop server, show log, show model config, edit config
• Running (no model) — load a profile, stop server, show log, edit config

When more than one instance is running, the relevant actions (stop, unload, show log) present an instance picker disambiguated by host:port.

Each profile row shows its title, the effective context size the backend will actually receive (compacted to 4K / 65K / 131K / 1M), a [server] tag when more than one backend is enabled, and the ★ favourite marker:

▸ DeepSeek Coder V2 Lite    65K  [LLaMA.cpp]
  Qwen 2.5 32B             131K  [LLaMA.cpp]
  reasoning-phi                  [Ollama   ]

The Ollama row is blank because Ollama's load request carries no context length — the column only shows what the backend is actually sent.

CLI commands

llama-launcher load <profile> [--restart]   # Activate a profile (no-op if already active; --restart forces)
llama-launcher unload [profile]             # Unload model from the matching instance
llama-launcher start [--profile p]          # Start server without a model (llamacpp requires --profile)
llama-launcher stop [target]                # Stop a server (target = host:port or backend name)
llama-launcher status [--json]              # Show all running instances (--json for structured output)
llama-launcher list [--json]                # List available profiles (--json for structured output)
llama-launcher logs [target] [-f]           # Tail an instance's log
llama-launcher logs clean [--days N|--all]  # Remove old log files
llama-launcher config validate              # Check config file for errors
llama-launcher config init [--force]        # Generate example config (--force overwrites)
llama-launcher config reset                 # Reset config to the example (overwrites)
llama-launcher version                      # Print version

A server that is still loading its model (llama.cpp answers its health endpoint with 503 for the whole load) is a first-class instance: status and the interactive menu show it as starting…, and stop / unload can target it. A plain load refuses to displace a still-loading server so a mistyped command cannot throw away a long model load; pass --restart to stop and replace it (ADR-0010).

When the port is already taken

Starting a llama.cpp server checks the target port first, and refuses before forking if another process is listening there:

Error: port 1111 is already in use — LLaMA.cpp cannot bind 0.0.0.0:1111
Listening now:
  PID 15481 (llama-server)
  PID 62070 (Code Helper (Plugin))
Stop the occupying process, or give this profile a different `port` — in its own section or under `defaults`

Any interface counts: a process holding 127.0.0.1:<port> blocks a 0.0.0.0 bind just as a wildcard listener does. That case is worth knowing about, because a foreign listener on loopback also shadows your server — the launcher's health probes reach the squatter instead, so status reports nothing running and every command waits out a timeout before answering. Editors and IDEs are a common source: VS Code's Remote-SSH and Dev Containers automatically forward ports from the remote machine onto the same local port number, which will quietly take over a port your local server already owns. lsof -nP -iTCP:<port> -sTCP:LISTEN shows every holder.

Remote control from a container (MCP)

llama-launcher itself has no network surface — it is a one-shot CLI (ADR-0002). When a client on another machine needs to control which model is running — typically a coding agent in a container reaching back to the host — an optional, separate binary, llama-launcher-mcp, exposes the lifecycle commands as MCP tools over HTTP. It runs on the host, implements every tool by shelling out to the CLI, and never proxies inference traffic (ADR-0008).

Tools: list_profiles, server_status, tail_log (read) and load_profile, unload_model, start_server, stop_server (mutating, omitted under --read-only).

Trust model: access is gated by a source-IP allowlist, not a token. Bind the listener to the host's container-facing bridge interface (not 0.0.0.0) and allow the container — by IP, CIDR, hostname, or simply by naming the bridge interface so any IP the bridge hands the container is covered. The client receives no secret it could leak — appropriate when the remote is a cloud LLM agent you don't want to hand credentials to.

llama-launcher-mcp --listen 192.168.64.1:7331 --allow-interface bridge100
#   --listen           container-facing bridge IP:port (not 0.0.0.0)
#   --allow-interface  local interface whose subnet(s) to allow; repeatable
#   --allow            client IP, CIDR, or hostname; repeatable; loopback by
#                      default. Hostnames resolve once at startup — prefer
#                      --allow-interface or a private CIDR (192.168.64.0/24).
#   --llama-launcher-bin  path to the CLI (default: PATH lookup)
#   --config           llama-launcher config path, forwarded to each call
#   --read-only        expose only the read tools

Then point the container's MCP client at http://192.168.64.1:7331/mcp — no token, just the URL.

Using llama-launcher as a Go library

The launcher is importable as well as runnable: the launcher/ package is a curated Go API over the same core, so your program can load a profile, see what is running, and stop or unload it in-process instead of shelling out to the CLI (ADR-0011, TDD §16).

go get github.com/airiclenz/llama-launcher/launcher@v1.7.0

cfg, err := launcher.LoadConfig(launcher.DefaultConfigPath(), func(warning string) {
	log.Printf("config warning: %s", warning)
})
if err != nil {
	return err
}
profile, err := cfg.ResolveProfile("qwen-coder") // merged with defaults, model path resolved
if err != nil {
	return err
}
go func() { // the lifecycle verbs block — run them off your UI goroutine
	inst, started, err := launcher.LoadProfile(cfg, profile, false,
		func(step string) { log.Printf("step: %s", step) },       // progress
		func(notice string) { log.Printf("notice: %s", notice) }, // e.g. the drift notice
	)
	if err != nil {
		log.Printf("load failed: %v", err)
		return
	}
	log.Printf("%s serving at %s (server started: %v)", profile.Name, inst.Addr(), started)

	// Stop is also how you cancel a load that is still in flight.
	if _, err := launcher.Stop(inst.Addr()); err != nil {
		log.Printf("stop failed: %v", err)
	}
}()

The rest of the surface is DiscoverRunningInstances(cfg), Unload(backend, addr), and four sentinels for errors.Is: ErrConfigNotFound, ErrNotRunning, ErrStartupTimeout and ErrUnsupported. The library never writes to your stderr — warnings arrive through the callbacks, and a nil callback discards them.

ErrStartupTimeout is the one worth handling explicitly: it means the activation wait expired, not that the load failed. The launcher deliberately leaves the server running, so treat it as "not yet" and keep watching the address with DiscoverRunningInstances.

Two caveats before you wire it in: there is one config per process (per-server API keys land on a process-global backend registry, so the last LoadConfig wins), and the lifecycle verbs block — activation waits up to ~30 seconds, plus up to ~20 more when a restart has to stop the current occupant — so call them from a goroutine and serialize your own calls against the same address.

Supported platforms

The package compiles on macOS, Linux and Windows, and each verb works wherever its mechanism exists (ADR-0012). No build tags, no stubs on your side.

Platform	What you get
macOS	Everything.
Linux	Everything. (Only the TUI's macOS-specific memory readout is dropped.)
Windows	Everything the launcher drives over HTTP: discovery, model load/unload against Ollama or LM Studio, and LoadProfile against a server that is already running. LM Studio start/stop works via the lms CLI. Starting llama-server or ollama serve is refused (wrapping ErrUnsupported) because Windows lacks the unix process control the launcher would need to stop it again.

Building

Requires Go 1.26+.

make build             # Build the binary (for local testing)
make build-mcp         # Build the optional MCP control-plane adapter
make test              # Unit tests (go test ./...)
make cross             # Cross-compile gate: build + vet for darwin/linux/windows
make check             # test + cross — run this before committing; starts no process
make test-integration  # Real-backend integration suite (host only; see below)
make test-all          # Both test layers (host only)
make clean             # Remove the binaries

make cross is the platform contract as a check (ADR-0012): it builds and vets the whole tree — test files included — for macOS, Linux and Windows, so a portability regression fails here instead of in an importing client's CI.

make test-integration starts and stops real servers (llama-server, Ollama, LM Studio) on the machine running it — run it manually on the host, never in CI or a container. Each test skips when its backend binary is not on PATH. Set INTEGRATION_MODEL_LLAMACPP (absolute .gguf path), INTEGRATION_MODEL_OLLAMA, and/or INTEGRATION_MODEL_LMSTUDIO to exercise the model load/unload steps.

The version is read from the VERSION file and injected at build time.

Architecture

All code lives in internal/launcher/, with the public launcher/ package a thin facade over it. Three LLM servers are implemented behind a common LLMServer interface: llama.cpp, Ollama, and LM Studio. The optional MCP adapter is a separate binary under cmd/llama-launcher-mcp/ and is the only component with a network listener.

The launcher does not persist runtime state. Each command rediscovers running servers by probing the addresses in your config and asking each server's own API which model is loaded. llama-launcher logs covers launcher-managed servers only.

The architectural decisions are written down as ADRs; the domain language is in CONTEXT.md; the technical design doc is llama-launcher.TDD.md.

Path	Purpose
~/.config/llama-launcher/config.yaml	Configuration
~/.config/llama-launcher/logs/	Server log files for instances the launcher started

License

MIT