codex-prune-proxy

README

codex-prune-proxy

A local proxy that sits between codex and its backend and lets the model itself drop tool outputs it is finished with, so they stop being re-sent on every later turn.

No config file to edit. No extra API key. No changes to codex.

go install github.com/hoveychen/codex-prune-proxy@latest
codex-prune-proxy &

codex exec \
  -c model_provider=prune \
  -c model_providers.prune.name=Prune \
  -c model_providers.prune.base_url=http://127.0.0.1:8790 \
  -c model_providers.prune.wire_api=responses \
  -c model_providers.prune.requires_openai_auth=true \
  -c model_providers.prune.supports_websockets=false \
  "research task here"

curl -s localhost:8790/_prune/stats | jq

Single static binary, no dependencies beyond the Go standard library.

---

The problem

Prompt caching is a prefix match: an agent's conversation grows append-only, so every turn is "read the old prefix, write the new tail". A tool output you no longer need does not go away — it keeps being re-read on every subsequent turn, forever, at the cached-input price.

The worst offender is the probe: the model runs rg/ls/cat across a tree to find out one thing, gets back a few hundred lines, extracts the one fact it needed, and the other 99% of that output stays in the prefix for the rest of the session.

What this does

codex ──(-c model_provider=prune)──▶ prune-proxy ──(headers forwarded verbatim)──▶ backend
                                          │
                                          ├─ appends a short inventory of last turn's
                                          │  big outputs to the END of the input array
                                          ├─ strips the model's trailing PRUNE:/KEEP:
                                          │  lines out of the response stream
                                          └─ replaces marked outputs with a placeholder
                                             on every later request

The model ends a reply with one line per decision:

PRUNE: a1b2
KEEP: c3d4 "fn spawn_session" "impl Drop for Session"

PRUNE drops the whole output. KEEP keeps only the lines containing any of the given substrings and drops the rest — a grep, chosen by the model, over its own tool output.

The pruned output is replaced by a placeholder that preserves the original command, so the model knows it already ran this and can re-run it if it turns out to need the full text:

[output pruned by harness · 340 lines · cmd: `rg -n "fn spawn" src/` · rerun to get it back]

Design decisions (and why)

The model decides, never the proxy. The proxy has no semantic information about whether content is still useful, and the cost of guessing wrong (deleting evidence the model still needs) is much higher than the 0.1× cached read it saves. There is no heuristic mode.

No token counts, no context budget, no "age" in the inventory. The inventory carries only identity: a short id, the original command, and a line count. Showing the model how many tokens each output costs turns the question from "do I still need this?" into "which one saves the most?" — a different and worse question. The line count stays because it is a locating signal (a 340-line output hints that KEEP might be the better move), not a cost signal.

The inventory goes at the very end of the input array. The tool definitions and system blocks sit at the front of the prefix; touching anything there invalidates every cached token after it. Appending at the tail costs only the tail.

Marks are stripped from the response stream. codex never sees the PRUNE:/KEEP: lines — not in its UI, not in its rollout, and therefore not in the next request either. The protocol costs zero prefix tokens. If the stripping state machine is ever unsure, it forwards the original bytes: leaking a marker is acceptable, corrupting the reply is not.

Marks are not carried by a tool call. codex would not recognise a proxy-injected tool, so the proxy would have to intercept the call, synthesise a result, and make another upstream request — and in that extra round trip the model regenerates its entire answer at full output price. That structurally eats the saving. Text marks ride the tail of the same response for a few dozen output tokens.

Only outputs ≥ PRUNE_MIN_OUTPUT_CHARS are offered. This is a gate on our spend, not a judgement about content: an inventory line costs full-price uncached tokens every turn, so it is not worth spending one on an output too small to pay it back.

When it pays off, and when it does not

Measured with exact per-mark accounting (save = removed_tokens × benefiting_turns × 0.1, cost = inventory_turns × ~350 at full price). Small sample — treat as direction, not precision.

task shape	runs	outcome
Exploratory research — open-ended investigation, no starting list, lots of dead ends	4	4/4 net positive, median +17.7k equivalent input tokens, median 48k tokens removed per run
Targeted code reading — well-specified questions, precise probes, 6–9 turns	3	net negative in aggregate; outputs were mostly still needed later

Net gain scaled roughly linearly with session length across the research runs (≈ +5k equivalent tokens per extra turn, break-even around 7 turns). The longest run measured was 13 turns (+26.5k, see below); beyond that it is untested.

So: turn it on for open-ended research and investigation. Leave it off for short, well-specified tasks. In the targeted runs the model correctly declined to mark almost anything — the outputs really were still needed — and the inventory cost was pure loss.

Quality was unaffected in every comparison run (identical scores on an automatically-graded 8-point rubric), which is the one thing that has to hold before any of the cost arithmetic matters.

A real session

Investigating this repository with pruning on — 13 turns, exact ledger from /_prune/stats:

 id    mode   marked   removed / original    benefiting turns   save
 ZY2Y  prune  turn 4    9,958 / 10,038 tok          9          8,962
 IkKp  keep   turn 5    5,704 /  5,999             8          4,563
 gvE9  keep   turn 6    5,356 /  5,567             7          3,749
 lfD6  keep   turn 7    8,700 /  8,870             6          5,220
 LFm7  prune  turn 8    6,728 /  6,783             5          3,364
 Y5UF  keep   turn 9    5,773 /  5,950             4          2,309
 dZ64  prune  turn 10   3,603 /  3,662             3          1,081
 zh1s  prune  turn 11   3,809 /  3,871             2            762
 Atcv  prune  turn 13         —  never applied (marked on the last turn)

 save 30,011   inventory cost 3,500   NET +26,511 equivalent input tokens

Note the shape: the first mark lands on turn 4 and earns the most because it has the most turns left to benefit from; the last one lands on the final turn and earns nothing. KEEP was chosen 4 times out of 9 — on file dumps where a handful of lines mattered and the rest did not.

Behaviour worth knowing

• The first mark lands early. In the research runs it was consistently on the third turn — the first turn a large output existed. The model reacts to a big output immediately; it does not wait until the end.
• Marks on the final turn are wasted. There is no later request to apply them to. Roughly 1 in 8 marks landed there.
• KEEP gets used once the outputs are file dumps. In the 13-turn run above it was chosen 4 times out of 9; in shorter, more targeted runs it never appeared. When used, the model picked out a handful of load-bearing lines and dropped the rest.
• Wording did not matter. Rewriting the instruction from conservative ("keep it if you may still need it") to permissive ("default to dropping") changed nothing measurable.
• The stronger model marks more. Faster/cheaper models in the same family marked less often, and sometimes marked small outputs while ignoring the one large one.

Limitations

• Small sample. Single model family, session lengths 6–12 turns, a handful of runs per condition.
• End-to-end A/B is hopeless at this effect size. Session-to-session variance in total token spend (same task, same model) reached 60–95%, while the expected effect is a few thousand tokens. Resolving it end-to-end would need on the order of 100+ runs per arm. Judge this by the per-mark ledger, not by comparing two total bills.
• The price constants are assumptions. 0.1× for a cached read and full price for the inventory are the model used in ledger(); adjust them for your provider.
• Session state is in-memory. Losing it just means a turn goes un-pruned.
• Only custom_tool_call_output items are rewritten. Everything else — reasoning blocks (encrypted), tool definitions, prompt_cache_key — is forwarded byte-for-byte.

How it works, in more detail

See docs/DESIGN.md for the wire-level details: what codex actually puts on the wire, why the proxy can be stateless, which SSE events carry the text that has to be cleaned, and the full protocol grammar.

Tests

go test ./...

Covers directive parsing, body integrity across 300 randomised stream-chunk boundaries, the mid-reply false-positive case, id suffix matching, both command-literal styles, the placeholder rewrite, KEEP line selection, the size gate, the ledger arithmetic, passthrough mode, unknown-field preservation, and usage accounting.

A Python prototype came first; it lives in the git history at f102fc3 if you want the smaller reference implementation.

License

MIT

Documentation

Overview

codex-prune-proxy — a Responses API proxy that lets the model drop its own spent tool outputs, so they stop being re-sent on every later turn.

The judgment is entirely the model's. The proxy only supplies an inventory of what could be dropped, executes the decisions the model states, and keeps a per-mark ledger. There is no heuristic mode and no extra model call.