context-guru

README

context-guru

Provider-agnostic context engineering for LLM agents. Shrink the tokens every request carries — losslessly, or lossy-but-reversibly — without touching the agent.

Quickstart · Why it wins · Components · Docs · Reproduce

---

context-guru is a single Go core that reduces the token cost of LLM-agent traffic. The same core runs as an HTTP proxy/gateway (drop-in, any language, zero agent changes) or as an in-process plugin. It operates on the messages array — dropping redundant tool output, collapsing superseded runs, projecting large reads down to what's relevant — and every reduction is safe by construction:

• Fail open, always — any component error or panic reverts that component only; the original request is always a valid fallback.
• Never worse — a component that would grow a message is reverted. You never pay to compact.
• Reversible — every lossy drop leaves a <<cg:HASH>> marker and stashes the original, recoverable via a model-callable context_guru_expand tool or GET /expand.

Benchmark: the cheapest & highest-reward arm on SWE-bench Verified

Evaluated live, end-to-end, with the claude-code agent on aws/claude-sonnet-5, against a no-compaction baseline, against the headroom request-stream proxy, and against rtk (Rust Token Killer, a shell-level Bash-output hook). All 50 tasks scored under all four arms.

dimension	baseline	context-guru	headroom	rtk
tasks solved	86%	88%	80%	86%
total billed cost vs baseline	—	−13.2%	−5.3%	−9.0%
cache-read tokens vs baseline	—	−17.8%	−6.3%	−10.8%
cache-write tokens vs baseline	—	−0.4%	−0.9%	−1.1%
mean steps / task vs baseline	—	−13.9%	−2.8%	−8.0%
added latency / req	—	117 ms	63 ms	0 ms
tool's own LLM cost	—	$0.31	$0	$0

context-guru is the cheapest arm and solves the most tasks — it cuts billed cost 13.2% vs no compaction, driven by an 17.8% cache-read reduction, while keeping cache-write within 1% of baseline (it never busts the cache). It does this by freezing each compaction and replaying it byte-identically every turn, so the saving compounds across the whole session. The surprise is rtk: a simple deterministic shell filter is the 2nd-cheapest arm (−9.0%), reward-neutral (86% = baseline), at zero request-path latency and $0 tool cost — it beats the headroom proxy on both cost and reward. rtk's ceiling is that it only compresses Bash-tool output (Claude Code's built-in Read/Grep/Glob bypass its hook), which is why the whole-request proxy goes deeper. Full four-way study, per-task/per-component breakdowns, real before→after examples, and how to reproduce: docs/RESULTS.md.

Architecture

flowchart LR
  A[Agent] -->|chat request| H{Host adapter}
  H -->|proxy: proxy.Handler| P[apply.Body]
  H -->|in-process: AuthBridge plugin| P
  P -->|messages array| PIPE[Pipeline<br/>ordered components]
  PIPE --> P
  P -->|byte-lossless splice| UP[Upstream provider]
  UP -->|response| EX[expand loop]
  EX -->|resolve markers from Store| UP
  EX --> A
  PIPE -.per-component Report.-> M[Emitter / Aggregator]
  PIPE -.stash originals.-> S[(Store<br/>TTL+LRU)]
  EX -.resolve.-> S

Components implement one of two lossiness-typed interfaces and are stacked in config order:

flowchart TD
  C["Component — Name() · Enabled(ctx)"]
  C --> R["Reformat: lossless repack<br/>format · toon · cacheinject"]
  C --> O["Offload: drop + stash, returns cache_keys<br/>skeleton · dedup · collapse · failed_run<br/>cmdfilter · extract · extract_llm · smartcrush · mask · summarize"]

Install

Requires Go 1.26 and a C toolchain (CGO_ENABLED=1). Build from the repo root:

CGO_ENABLED=1 go build -tags cg_skeleton -o bin/context-guru-proxy ./cmd/context-guru-proxy

The cg_skeleton build tag pulls in tree-sitter (via cgo) so the skeleton component can parse code. It is optional — omit the tag and the tree-sitter dependency for a pure-Go build; the skeleton component is simply inert without it, everything else works. Or build the gateway image (see docs/setup.md):

docker build -t context-guru:local .

Quickstart (60 seconds)

# 1 — run the proxy (ships with the SWE-bench-winning cache-aware config by default)
./bin/context-guru-proxy                          # --preset codesmart; listens on :4000 (LISTEN_ADDR to change)

# 2 — point any agent at it (one port serves both dialects)
export ANTHROPIC_BASE_URL=http://localhost:4000/anthropic
export OPENAI_BASE_URL=http://localhost:4000/openai/v1
claude                                            # e.g. Claude Code

# 3 — watch the savings add up
curl -s localhost:4000/stats | jq                 # token-weighted savings rollup

Or drive it directly with an Anthropic-style request (this is exactly how the quickstart is tested — see docs/get-started/quickstart-proxy.md):

curl -s localhost:4000/anthropic/v1/messages \
  -H 'content-type: application/json' \
  -H "Authorization: Bearer $YOUR_KEY" \
  -d '{"model":"...","max_tokens":64,"messages":[ ... ]}'

Presets: codesmart (the default — the SWE-bench-winning cache-aware config [format, dedup, failed_run, cmdfilter, extract_llm, extract, cacheinject]), codesafe (the same minus the LLM pass — deterministic-only [format, dedup, failed_run, cmdfilter, extract, collapse, cacheinject], zero model calls by policy), plus general, agent, coding, mcp, balanced, safe, summarize, off. codesmart's LLM relevance-trimmer (extract_llm) engages only when a cheap model is configured (CHEAP_MODEL*); without one it safely no-ops and behaves like codesafe. See docs/components.md and docs/reference/presets.md.

Flag / env	Default	Purpose
--preset / PRESET	codesmart	pipeline preset when no --config
--config / CONFIG	—	YAML config (overrides preset)
LISTEN_ADDR	:4000	listen address
--anthropic-upstream / ANTHROPIC_UPSTREAM	https://api.anthropic.com	Anthropic upstream base
--openai-upstream / OPENAI_UPSTREAM	https://api.openai.com	OpenAI upstream base
OPENAI_API_KEY / ANTHROPIC_API_KEY	—	real key injected on forward (gateway mode); empty = pass client auth through
CHEAP_MODEL (+ CHEAP_MODEL_*)	—	dedicated cheap model for the LLM components (extract_llm, summarize)
FORCE_MODEL	—	overwrite the request model (eval-containers EVAL_MODEL)

Routes: POST /openai/v1/chat/completions, POST /anthropic/v1/messages, GET /healthz, GET /stats (savings rollups), GET /expand?id= (recover an offloaded original). Per-request: header x-context-guru-session sets the session key; x-context-guru-bypass: true skips the pipeline.

The pipeline

Every component operates on tool-output messages. Reformat = lossless repack; Offload = drop bytes, stash the original, leave a recoverable marker. Real, live-captured before→after examples for each are in docs/components.md and docs/results/components.md.

Component	Kind	What it does
format	Reformat	re-encodes pretty JSON tool output as compact JSON
toon	Reformat	re-encodes a uniform JSON array as TOON (header once, one row per item)
cacheinject	Reformat	adds an Anthropic cache_control breakpoint on a stable prefix boundary
dedup	Offload	replaces a byte-identical earlier tool output with a pointer
failed_run	Offload	collapses superseded test/build runs, keeps the latest in full
cmdfilter	Offload	shrinks structured command output via declarative DSL filters
extract	Offload	deterministic noise collapse (repeated lines, blank runs, progress bars)
extract_llm	Offload (LLM)	a cheap model writes a sandboxed filter that trims to what's relevant
collapse	Offload	head/tail window on any oversized output (last-resort fallback)
mask	Offload	age-based GC — keep the newest N tool outputs, stash older ones
skeleton	Offload	replaces code-block function bodies with signatures (needs cg_skeleton)
smartcrush	Offload	keeps anchor items of a long JSON array, drops the middle
summarize	Offload (LLM)	compresses the middle of the trajectory into one summary (run alone)

Integrate

Option	What	Where
Proxy / gateway	context-guru-proxy in front of the provider; the eval-containers gateway image	proxy/, cmd/context-guru-proxy/
In-process plugin	AuthBridge (Rossoctl sidecar) plugin importing this module, running the same pipeline on pctx.Body	plugin lives in cortex; reuses apply.Body + expand/
(also) bifrost LLMPlugin	run the pipeline as a PreRequestHook inside any bifrost deployment	adapters/bifrost/

Details in docs/integrations.md.

Docs

• docs/design.md — architecture: component model, fail-open pipeline, store, session, expand loop, metrics.
• docs/components.md — every registered component: how it works, live before→after, lossiness, config, best use.
• docs/integrations.md — proxy gateway vs AuthBridge plugin, with request paths.
• docs/setup.md — setup + a concrete SWE-bench run through the eval-containers gateway.
• docs/RESULTS.md — the live four-way SWE-bench Verified benchmark (Claude Code, aws/claude-sonnet-5): context-guru is the cheapest arm (−13.2% billed cost vs baseline) and solves the most tasks (88%); headroom −5.3%/80%; rtk (shell-level Bash-output hook) −9.0%/86% at $0 tool cost.

License

Apache-2.0. See LICENSE. A Rossoctl platform component.