raggate

README

raggate

日本語 README

Regression tests and quality gates for LLM/RAG pipelines. Define test cases in YAML, run them against your endpoints, and fail your CI when answer quality drops, latency regresses, or cost jumps — before your users notice.

raggate run -c suite.yaml -o results.json
raggate compare baseline.json results.json   # exit 3 when the gate fails

• Single Go binary, no framework, one dependency (yaml.v3)
• Works with OpenAI-compatible APIs, the Anthropic API, and any HTTP RAG endpoint
• Deterministic graders (exact / contains / regex / JSON Schema subset), retrieval metrics (Recall@k / MRR against returned citations), and an opt-in LLM judge
• Reports quality score, p50/p95 latency, and token cost per endpoint — as Markdown, JSON, or JUnit XML
• compare diffs two runs and enforces thresholds, designed to gate pull requests (model upgrades, prompt changes, retriever tweaks)

Why

Shipping a RAG pipeline is easy; keeping it good is not. Model updates, prompt edits, and index changes silently change answers. raggate treats answer quality like any other regression: version a golden dataset, run it in CI, and block the merge when quality drops.

Quickstart

go install github.com/mutton-dev/raggate/cmd/raggate@latest

# 1. start the bundled mock RAG server (or point at your own endpoint)
go run github.com/mutton-dev/raggate/examples/mockrag@latest --port 8080 &

# 2. write a starter suite
raggate init

# 3. record a baseline
raggate run -c suite.yaml -o baseline.json

# 4. after changing your pipeline, run again and compare
raggate run -c suite.yaml -o results.json
raggate compare baseline.json results.json
echo $?   # 0 = gate passed, 3 = quality gate failed

Try --degrade on the mock server to see the gate fire:

go run github.com/mutton-dev/raggate/examples/mockrag@latest --port 8080 --degrade

Suite configuration

name: my-rag

endpoints:
  - name: prod-rag
    kind: http_rag           # openai | anthropic | http_rag
    base_url: https://rag.example.com/query
  - name: gpt
    kind: openai
    base_url: https://api.openai.com
    model: gpt-4o-mini
    api_key_env: OPENAI_API_KEY   # env var NAME — keys never live in YAML

graders:                     # suite-wide; override per case with `graders:`
  - type: contains
  - type: recall_at_k
    k: 3

thresholds:                  # enforced by `compare` (zero value = not enforced)
  min_score: 0.7
  max_score_drop: 0.05
  max_p95_ms: 3000
  max_cost_increase_pct: 20

pricing:                     # JPY per 1M tokens, used for cost accounting
  - model: gpt-4o-mini
    input_per_1m: 22
    output_per_1m: 90

cases:
  - id: refund-policy
    input: "What is the refund window?"
    expected:
      contains: ["30 days"]
      doc_ids: ["policy-refunds"]     # matched against returned citations

Grader types: exact, contains, regex, json_schema (subset: type/required/properties/enum), recall_at_k, mrr, llm_judge (set judge_endpoint to one of your endpoints).

http_rag endpoints receive POST {"query": "..."} and should reply {"answer": "...", "contexts": [...], "citations": ["doc-id", ...]}.

CI integration

See .github/workflows/gate-example.yml for a complete PR-gating workflow that posts the compare report as a PR comment. Exit codes: 0 pass, 1 runtime error, 3 gate failed.

Notes for hardening your gate:

• compare enforces the baseline's thresholds by default (--thresholds current to opt out), so a PR cannot quietly relax its own gate.
• Keep baseline.json on the base branch (or fetch it from a trusted store); a PR that regenerates its own baseline defeats the purpose.
• run exits non-zero when any case fails to execute (--allow-case-errors to tolerate partial outages).

Security notes

• API keys are only ever read from environment variables named in the YAML.
• Endpoints must be http/https. raggate does not further restrict private/link-local addresses: treat suite files as code and do not run untrusted suites in privileged CI environments.
• Response bodies are capped (32 MiB) to keep runners safe from oversized replies.

Roadmap (v0.2)

• Langfuse trace export
• LLM-judge cost/latency accounting
• Action SHA pinning in the example workflow
• Provider integration test harness against recorded fixtures

Development

go build ./... && go vet ./... && go test ./...

Built by three coding agents (Claude Code, Codex, Grok) working in parallel with package-level ownership, cross-review, and an integration gate — the same regression-first workflow raggate is designed to enforce.

License

MIT