tbg-ollama-swap-prompt-optimizer

README

TBG (O)llama Swap + Prompt Optimizer

Forked from Benson Wong's upstream project: mostlygeek/llama-swap.

TBG (O)llama Swap + Prompt Optimizer is focused on local agent workflows (for example Claude Code CLI) where prompts become very large, repetitive, and unstable for low-VRAM local inference.

What This Fork Adds

• Runtime per-model context override from UI/API
• Prompt optimization policies per model
• Latest optimization snapshot endpoint for audit/debug
• Optional Ollama discovery and passthrough hook
• Unified model list for configured llama.cpp models and live Ollama models
• Chat attachments in Playground (+ upload for files/images with drag-and-drop)
• Source link badges in assistant messages for tool/web results
• Activity prompt-flow preview (current user turn + agent steps)

Why This Exists

Long local coding-agent sessions can produce:

• repeated transcript blocks
• context overflow errors
• wasted context budget on low-value history

TBG (O)llama Swap optimizes request payloads before forwarding upstream so context is used for relevant content.

How TBG (O)llama Swap Works

Request path:

Client (Claude Code / OpenWebUI) -> TBG (O)llama Swap + Prompt Optimizer -> llama.cpp or Ollama

Detailed behavior:

1. Client calls an OpenAI-compatible endpoint with a model.
2. The proxy resolves the target model source:

• configured model in config.yaml
• discovered live model with ID ollama/<name>

1. If needed, the proxy swaps upstream process state to match the requested configured model.
2. Runtime model settings are applied:

• context override (/api/model/:model/ctxsize)
• prompt optimization policy (/api/model/:model/prompt-optimization)

1. Prompt payload is optionally compacted based on policy:

• off (no optimization)
• limit_only (optimize only near/over limit)
• always (aggressive optimization)
• llm_assisted (smart optimization using model-assisted summarization)

1. Request is forwarded and response is streamed back to client.
2. Latest optimization result can be inspected via:

• /api/model/:model/prompt-optimization/latest

Prompt Optimization Benchmarks

Test environment:

• WSL Ubuntu
• model: gpt-oss-20b-F16
• context: 8096
• payload: huge repetitive coding-session transcript

Mode naming used in this README:

• off = optimization disabled
• smart = internal policy limit_only
• aggressive = internal policy always
• llm_assisted = model-assisted summarization mode (separate from smart)

Latest Same-Payload Test (smart vs aggressive)

Mode	prompt_tokens	completion_tokens	total_tokens	latency
smart (limit_only)	83	32	115	~2415 ms
aggressive (always)	108	32	140	~2323 ms

Interpretation:

• both modes stayed inside context and completed successfully.
• in this test, smart produced fewer prompt tokens than aggressive.
• aggressive is not guaranteed to be smaller on every prompt; outcomes depend on repetition shape and downstream tokenization.

Historical Project Runs (aggressive vs llm_assisted)

Metric	aggressive (always)	llm_assisted
Avg latency (repeated prompt A/B)	14.164s	13.832s
Avg latency (quality test runs)	5.949s	5.738s
Quality score (marker recall, avg)	6.0/6	6.0/6

Off Baseline Status

• off vs aggressive baseline values were tested earlier but numeric logs are not currently retained in-repo.

200k Coding Looseness Factor (Practical Rule)

• off: 0% intentional loss, highest fidelity, highest overflow risk.
• smart (limit_only): typically ~1-5% semantic loss risk on real coding threads.
• aggressive (always): typically ~5-15% semantic/detail loss risk.

Recommended looseness factor for coding at 200k:

• smart: 1.05
• aggressive: 1.15

Operational guidance:

• default to smart (limit_only) for long coding sessions.
• use aggressive (always) only when repetitive bloat still pushes context pressure.
• keep critical constraints in the latest system/user message so they survive compaction.

UI

Playground highlights:

• icon-only send/stop controls
• single + button for file/image attachments
• drag-and-drop file support into chat input
• attachment chips with remove action
• edit and delete actions on user turns
• deleting a user turn also deletes its paired assistant reply

Activity highlights:

• request metrics table (existing)
• latest prompt-flow panel:
  • shows preview of prompt payload sent to model
  • labels each request as New User Request or Agent Step
  • keeps steps for current user turn and auto-resets on next user request

Models panel highlights:

• Kill llama.cpp emergency stop
• Refresh Config reloads config from disk without full app restart
• Restart TBG (O)llama Swap performs soft restart (stop models + reload runtime config/state)

Installation Guides

Common requirements (all platforms)

• Git
• Go 1.24+
• Node.js 20+ and npm (for UI build)
• Optional runtime backends:
  • llama.cpp (llama-server)
  • ollama

Clone once:

git clone https://github.com/Ltamann/tbg-ollama-swap-prompt-optimizer.git
cd tbg-ollama-swap-prompt-optimizer

Linux (Ubuntu/Debian)

Install dependencies:

sudo apt update
sudo apt install -y git make curl ca-certificates build-essential

Install Node.js 20:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

Install Go 1.24+ (from distro or official tarball), then verify:

go version
node -v
npm -v

Build:

make clean linux

Run:

./build/llama-swap-linux-amd64 -config ./config.yaml -listen 0.0.0.0:8080

macOS (Apple Silicon)

Install dependencies:

xcode-select --install
brew install git go node make

Verify:

go version
node -v
npm -v

Build:

make clean mac

Run:

./build/llama-swap-darwin-arm64 -config ./config.yaml -listen 0.0.0.0:8080

Windows (native PowerShell)

Install dependencies:

• Git for Windows
• Go 1.24+
• Node.js 20+ LTS
• Optional: make (or use the manual build commands below)

Verify:

git --version
go version
node -v
npm -v

Build UI:

cd ui-svelte
npm install
npm run build
cd ..

Build Windows binary:

go build -o .\build\llama-swap-windows-amd64.exe .

Run:

.\build\llama-swap-windows-amd64.exe -config .\config.yaml -listen 127.0.0.1:8080

Windows + WSL (recommended for llama.cpp workflows)

In Windows PowerShell, install WSL Ubuntu:

wsl --install -d Ubuntu

Inside WSL Ubuntu:

sudo apt update
sudo apt install -y git make curl ca-certificates build-essential
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs

Install Go 1.24+ in WSL, then build in WSL:

git clone https://github.com/Ltamann/tbg-ollama-swap-prompt-optimizer.git
cd tbg-ollama-swap-prompt-optimizer
make clean linux

Run in WSL:

./build/llama-swap-linux-amd64 -config ./config.yaml -listen 0.0.0.0:8080

Access UI from Windows browser:

• http://localhost:8080/ui/

Known-Good WSL Update + Restart Workflow

Use this exact sequence when you run the app from WSL but build from Windows workspace.

Build backend + UI in repo:

go build ./...
cd ui-svelte
npm install
npm run build
cd ..

Build Linux binary explicitly (important):

$env:GOOS='linux'
$env:GOARCH='amd64'
$env:CGO_ENABLED='0'
go build -o build/llama-swap-linux-amd64 .

Install into WSL:

$repoWslPath = wsl wslpath -a .
wsl -d Ubuntu -- bash -lc "install -Dm755 '$repoWslPath/build/llama-swap-linux-amd64' /home/admmin/bin/llama-swap"

Restart cleanly in WSL (detached):

wsl -d Ubuntu -- bash -lc 'pkill -9 -f llama-swap || true; pkill -9 -f llama-server || true; setsid /home/admmin/bin/llama-swap --config /home/admmin/llama-swap/config.yaml --listen 0.0.0.0:8080 >/tmp/llama-swap.log 2>&1 < /dev/null &'

Verify:

wsl -d Ubuntu -- bash -lc 'file /home/admmin/bin/llama-swap; ss -ltnp | sed -n "1p;/:8080/p"; curl -sS http://127.0.0.1:8080/health'

Expected:

• binary is ELF 64-bit (Linux), not Windows PE
• listener on *:8080
• health response OK

Quick Build Matrix

• make linux -> build/llama-swap-linux-amd64 and build/llama-swap-linux-arm64
• make mac -> build/llama-swap-darwin-arm64
• make windows -> build/llama-swap-windows-amd64.exe

Configuration (Full Model Sample)

healthCheckTimeout: 120
startPort: 10001
logLevel: info
includeAliasesInList: true
sendLoadingState: true

apiKeys:
  - ${env.LLAMASWAP_API_KEY}

macros:
  LLAMA_BIN: /home/admin/llama/cuda/llama.cpp/build/bin/llama-server
  MODEL_DIR: /home/admin/models
  HOST: 0.0.0.0
  THREADS: 16
  GPU_LAYERS: 999

hooks:
  on_startup:
    preload:
      - Qwen3-Coder-Next-MXFP4_MOE

models:
  Qwen3-Coder-Next-MXFP4_MOE:
    name: "Qwen3-Coder-Next"
    description: "Primary coding model"
    cmd: >
      ${LLAMA_BIN}
      --model ${MODEL_DIR}/Qwen3-Coder-Next-MXFP4_MOE.gguf
      --host ${HOST} --port ${PORT}
      --threads ${THREADS}
      --threads-batch 48
      --n-gpu-layers ${GPU_LAYERS}
      --n-cpu-moe 0
      --ctx-size 202752
      --batch-size 4096
      --ubatch-size 512
      --tensor-split 60,40
      --flash-attn on
      --jinja
      --parallel 1
    checkEndpoint: /health
    ttl: 0
    aliases:
      - tbg-coder-next
      - qwen3-coder-next
    useModelName: Qwen3-Coder-Next-MXFP4_MOE
    truncationMode: sliding_window
    filters:
      stripParams: "temperature,top_p,top_k,repeat_penalty"
      setParams:
        temperature: 0.7
        top_p: 0.95
        top_k: 40
        min_p: 0.01
    metadata:
      family: qwen3-coder-next
      quant: mxfp4
      vram_hint_gb: 24

  gpt-oss-20b-F16:
    name: "gpt-oss-20b"
    description: "Secondary reasoning/coding model"
    cmd: >
      ${LLAMA_BIN}
      --model ${MODEL_DIR}/gpt-oss-20b-F16.gguf
      --host ${HOST} --port ${PORT}
      --threads ${THREADS}
      --n-gpu-layers ${GPU_LAYERS}
      --ctx-size 262144
      --batch-size 2048
      --ubatch-size 512
      --flash-attn on
      --jinja
      --parallel 1
    checkEndpoint: /health
    ttl: 1800
    aliases:
      - gpt-oss
      - gpt-oss-20b
    truncationMode: strict_error

groups:
  coding:
    swap: true
    exclusive: true
    persistent: false
    members:
      - Qwen3-Coder-Next-MXFP4_MOE
      - gpt-oss-20b-F16

Runtime notes:

• Context can be overridden at runtime in UI and API (/api/model/:model/ctxsize).
• Prompt optimization policy is runtime-configurable per model.
• Ollama models are auto-discovered when Ollama is reachable and are shown as external/read-only for ctx override.

API Surface Added By This Fork

• POST /api/model/:model/ctxsize
• GET /api/model/:model/ctxsize
• POST /api/model/:model/prompt-optimization
• GET /api/model/:model/prompt-optimization
• GET /api/model/:model/prompt-optimization/latest
• GET /api/activity/prompts
• POST /api/config/reload
• POST /api/restart

Tool Runtime (HTTP + MCP)

This fork now includes a server-side tool runtime for OpenAI-style function-calling.

Flow:

1. Client sends /v1/chat/completions.
2. TBG (O)llama Swap injects enabled tool schemas into the request.
3. Model can emit tool_calls (or legacy function_call).
4. TBG executes calls (HTTP tools first class, MCP tools supported).
5. Tool outputs are appended as role=tool messages.
6. Model is called again until final answer or max tool rounds is reached.

Notes:

• Tools are orchestrated by TBG (O)llama Swap (not by llama.cpp alone).
• Works for clients that use OpenAI-compatible chat endpoints.
• Streaming requests are finalized as a synthetic SSE completion chunk in this MVP.
• When tool outputs contain URLs, source metadata is attached to assistant responses and rendered as clickable source badges in chat UI.

Tool Types

• http (example: searxng_web_search)
• mcp (example: Playwright MCP endpoint)

MCP argument behavior:

• For fixed MCP tools (remoteName set): model passes tool arguments directly.
• For MCP gateway tools (remoteName empty): model should pass:
  • name: remote MCP tool name (example: browser_navigate)
  • arguments: object for that remote tool
• HTTP tools support {query} and additional endpoint placeholders from tool arguments.

Tool Policies

Per tool:

• auto: model decides when to call tool
• always: force this tool as first choice
• never: do not expose/execute this tool

Global runtime settings:

• enabled: turn entire tool runtime on/off
• webSearchMode: off | auto | force
• maxToolRounds: loop cap for tool-calling iterations
• killPreviousOnSwap: stop previous ready llama.cpp model when swapping (default true)
• maxRunningModels: cap simultaneous ready models (default 1)
• requireApprovalHeader: require explicit approval header on requests
• approvalHeaderName: header key (default X-LlamaSwap-Tool-Approval)
• blockNonLocalEndpoints: block non-local tool endpoints for safer defaults

Tool Security Model (MVP)

• Local-only endpoint guard by default (localhost, loopback, host.docker.internal, .local).
• Optional per-tool requireApproval.
• Optional global approval header gate.
• Per-tool timeout control.
• Tool execution is audit-logged in proxy logs (name/type/duration/error status).

Playwright MCP Troubleshooting

If tool execution returns a browser launch error like:

• Chromium distribution 'chrome' is not found ...

install the expected browser channel in the runtime where MCP server runs:

npx playwright install chrome

If you run MCP in WSL, run the install in WSL. If you run MCP on Windows, run it on Windows.

Tool API Endpoints

• GET /api/tools
• POST /api/tools
• PUT /api/tools/:id
• DELETE /api/tools/:id
• GET /api/tools/settings
• PUT /api/tools/settings

Tool persistence:

• Stored next to config in tools.json
• Supports legacy array format and new structured format:
  • settings
  • tools

Notes

• This project focuses on practical local-model reliability for CLI agents with long repetitive prompts.
• For full base configuration options, see docs/configuration.md.