codex-prune-proxy

README

codex-prune-proxy

A local proxy that sits between codex and its backend and lets the model itself drop tool outputs it is finished with, so they stop being re-sent on every later turn.

No flags to remember. No extra API key. No changes to codex itself.

curl -fsSL https://raw.githubusercontent.com/hoveychen/codex-prune-proxy/main/install.sh | sh
codex-prune-proxy run

That is the whole setup. run points codex at the proxy by editing ~/.codex/config.toml, and puts the file back when you quit — so in any other terminal you keep using plain codex, with no flags and nothing to remember.

While it runs you get a monitor:

codex-prune-proxy v0.2.0
  codex config injected — plain `codex` goes through this proxy
  127.0.0.1:8790 → https://chatgpt.com/backend-api/codex   mode=model   up 7m12s
──────────────────────────────────────────────────────────────────────────────
  NET +25,859 equivalent input tokens
    saved 30,759   inventory cost 4,900   pruned 57,110 tokens from the prefix
    2 session(s)   19 turns   10 marks (9 applied)
    in 388,000 · cached 368,500 (95.0%) · out 16,000 · ≈100,500 equiv
──────────────────────────────────────────────────────────────────────────────
  session        turns marks applied     removed         net
  sess-a…aaaaa      13     9      8      49,633     +26,511
  sess-b…bbbbb       6     1      1       7,477        -652
──────────────────────────────────────────────────────────────────────────────
  mark   mode    turn     removed  turns        save
  Atcv   prune     13           0      0           0  wasted
  IkKp   keep       5       5,704      8       4,563
  ZY2Y   prune      4       9,958      9       8,962
──────────────────────────────────────────────────────────────────────────────
  Ctrl-C to quit — the codex config is restored on the way out

Other ways to install:

go install github.com/hoveychen/codex-prune-proxy@latest    # with a Go toolchain
# or grab a binary: https://github.com/hoveychen/codex-prune-proxy/releases/latest

Single static binary, no dependencies beyond the Go standard library.

---

The problem

Prompt caching is a prefix match: an agent's conversation grows append-only, so every turn is "read the old prefix, write the new tail". A tool output you no longer need does not go away — it keeps being re-read on every subsequent turn, forever, at the cached-input price.

The worst offender is the probe: the model runs rg/ls/cat across a tree to find out one thing, gets back a few hundred lines, extracts the one fact it needed, and the other 99% of that output stays in the prefix for the rest of the session.

What this does

codex ──(config.toml model_provider)──▶ prune-proxy ──(headers forwarded verbatim)──▶ backend
                                          │
                                          ├─ appends a short inventory of last turn's
                                          │  big outputs to the END of the input array
                                          ├─ strips the model's trailing PRUNE:/KEEP:
                                          │  lines out of the response stream
                                          └─ replaces marked outputs with a placeholder
                                             on every later request

The model ends a reply with one line per decision:

PRUNE: a1b2
KEEP: c3d4 "fn spawn_session" "impl Drop for Session"

PRUNE drops the whole output. KEEP keeps only the lines containing any of the given substrings and drops the rest — a grep, chosen by the model, over its own tool output.

The pruned output is replaced by a placeholder that preserves the original command, so the model knows it already ran this and can re-run it if it turns out to need the full text:

[output pruned by harness · 340 lines · cmd: `rg -n "fn spawn" src/` · rerun to get it back]

Commands

codex-prune-proxy run	bind, inject the codex config, show the monitor, restore on exit
codex-prune-proxy serve	just the proxy — touches no config, prints no UI (what a supervisor wants)
codex-prune-proxy stats	print the ledger of a running instance and exit
codex-prune-proxy restore	undo an injection left behind by a kill -9

Running it with no subcommand is the same as serve, so older invocations keep working.

About that config edit

Editing someone's config.toml is the kind of convenience that can ruin their day, so:

• the original is copied to config.toml.prune-backup before anything is written, and restoring prefers that byte-exact copy;
• a sentinel file records that an injection is live. kill -9 leaves it behind, and the next run reverts before backing up again — otherwise the backup would capture the injected file and the original would be gone for good;
• the port is bound before the config is touched, so a port clash cannot leave codex pointed at nothing;
• restore fixes it by hand at any time.

The edit itself is two sentinel-wrapped blocks: model_provider = "prune" above the first table header, and a [model_providers.prune] table at the end. An existing top-level model_provider is commented out in place rather than removed. See docs/DESIGN.md for why the position is load-bearing.

Flags

Each has an environment variable, so it also works over env or a systemd unit.

flag	env	default	what
-listen	PRUNE_LISTEN	127.0.0.1:8790	address to listen on
-upstream	PRUNE_UPSTREAM	https://chatgpt.com/backend-api/codex	the real backend
-mode	PRUNE_MODE	model	model, or off for a passthrough that still keeps books
-min-output-chars	PRUNE_MIN_OUTPUT_CHARS	2000	only offer outputs at least this big
-list-window	PRUNE_LIST_WINDOW	1	how many past turns of outputs to offer
-debug	PRUNE_DEBUG	on	log the inventory and marks to stderr (forced off under run)
-no-color	NO_COLOR	off	plain monitor output

CODEX_HOME is honoured for locating config.toml, so a sandboxed codex setup is injected in the right place.

Design decisions (and why)

The model decides, never the proxy. The proxy has no semantic information about whether content is still useful, and the cost of guessing wrong (deleting evidence the model still needs) is much higher than the 0.1× cached read it saves. There is no heuristic mode.

No token counts, no context budget, no "age" in the inventory. The inventory carries only identity: a short id, the original command, and a line count. Showing the model how many tokens each output costs turns the question from "do I still need this?" into "which one saves the most?" — a different and worse question. The line count stays because it is a locating signal (a 340-line output hints that KEEP might be the better move), not a cost signal.

The inventory goes at the very end of the input array. The tool definitions and system blocks sit at the front of the prefix; touching anything there invalidates every cached token after it. Appending at the tail costs only the tail.

Marks are stripped from the response stream. codex never sees the PRUNE:/KEEP: lines — not in its UI, not in its rollout, and therefore not in the next request either. The protocol costs zero prefix tokens. If the stripping state machine is ever unsure, it forwards the original bytes: leaking a marker is acceptable, corrupting the reply is not.

Marks are not carried by a tool call. codex would not recognise a proxy-injected tool, so the proxy would have to intercept the call, synthesise a result, and make another upstream request — and in that extra round trip the model regenerates its entire answer at full output price. That structurally eats the saving. Text marks ride the tail of the same response for a few dozen output tokens.

Only outputs ≥ PRUNE_MIN_OUTPUT_CHARS are offered. This is a gate on our spend, not a judgement about content: an inventory line costs full-price uncached tokens every turn, so it is not worth spending one on an output too small to pay it back.

When it pays off, and when it does not

Measured with exact per-mark accounting (save = removed_tokens × benefiting_turns × 0.1, cost = inventory_turns × ~350 at full price). Small sample — treat as direction, not precision. Full method, all runs and how to reproduce: docs/BENCHMARK.md.

task shape	runs	outcome
Exploratory research — open-ended investigation, no starting list, lots of dead ends	5	5/5 net positive, median +23.2k equivalent input tokens, median 49.6k tokens removed per run
Targeted code reading — well-specified questions, precise probes, 6–9 turns	3	net negative in aggregate; outputs were mostly still needed later

Net gain scaled roughly linearly with session length across the research runs (≈ +5k equivalent tokens per extra turn, break-even around 7 turns). The longest run measured was 13 turns (+26.5k, see below); beyond that it is untested.

So: turn it on for open-ended research and investigation. Leave it off for short, well-specified tasks. In the targeted runs the model correctly declined to mark almost anything — the outputs really were still needed — and the inventory cost was pure loss.

Quality was unaffected in every comparison run (identical scores on an automatically-graded 8-point rubric), which is the one thing that has to hold before any of the cost arithmetic matters.

A real session

Investigating this repository with pruning on — 13 turns, exact ledger from /_prune/stats:

 id    mode   marked   removed / original    benefiting turns   save
 ZY2Y  prune  turn 4    9,958 / 10,038 tok          9          8,962
 IkKp  keep   turn 5    5,704 /  5,999             8          4,563
 gvE9  keep   turn 6    5,356 /  5,567             7          3,749
 lfD6  keep   turn 7    8,700 /  8,870             6          5,220
 LFm7  prune  turn 8    6,728 /  6,783             5          3,364
 Y5UF  keep   turn 9    5,773 /  5,950             4          2,309
 dZ64  prune  turn 10   3,603 /  3,662             3          1,081
 zh1s  prune  turn 11   3,809 /  3,871             2            762
 Atcv  prune  turn 13         —  never applied (marked on the last turn)

 save 30,011   inventory cost 3,500   NET +26,511 equivalent input tokens

Note the shape: the first mark lands on turn 4 and earns the most because it has the most turns left to benefit from; the last one lands on the final turn and earns nothing. KEEP was chosen 4 times out of 9 — on file dumps where a handful of lines mattered and the rest did not.

Behaviour worth knowing

• The first mark lands early. Across the research runs it was always turn 3 or 4 — the first turn a large output existed. The model reacts to a big output immediately; it does not wait until the end.
• Marks on the final turn are wasted. There is no later request to apply them to. Roughly 1 in 8 marks landed there.
• KEEP gets used once the outputs are file dumps. In the 13-turn run above it was chosen 4 times out of 9; in shorter, more targeted runs it never appeared. When used, the model picked out a handful of load-bearing lines and dropped the rest.
• Wording did not matter. Rewriting the instruction from conservative ("keep it if you may still need it") to permissive ("default to dropping") changed nothing measurable.
• The stronger model marks more. Faster/cheaper models in the same family marked less often, and sometimes marked small outputs while ignoring the one large one.

Limitations

• Small sample. Single model family, session lengths 6–12 turns, a handful of runs per condition.
• End-to-end A/B is hopeless at this effect size. Session-to-session variance in total token spend (same task, same model) reached 60–95%, while the expected effect is a few thousand tokens. Resolving it end-to-end would need ~134 runs per arm (the arithmetic). Judge this by the per-mark ledger, not by comparing two total bills.
• The price constants are assumptions. 0.1× for a cached read and full price for the inventory are the model used in ledger(); adjust them for your provider.
• Session state is in-memory. Losing it just means a turn goes un-pruned.
• Only custom_tool_call_output items are rewritten. Everything else — reasoning blocks (encrypted), tool definitions, prompt_cache_key — is forwarded byte-for-byte.

How it works, in more detail

See docs/DESIGN.md for the wire-level details: what codex actually puts on the wire, why the proxy can be stateless, which SSE events carry the text that has to be cleaned, and the full protocol grammar.

See docs/BENCHMARK.md for how the numbers above were produced, why an end-to-end A/B cannot resolve this effect, and task templates to re-run it on a repository of your own.

Tests

go test ./...

Covers directive parsing, body integrity across 300 randomised stream-chunk boundaries, the mid-reply false-positive case, id suffix matching, both command-literal styles, the placeholder rewrite, KEEP line selection, the size gate, the ledger arithmetic, passthrough mode, unknown-field preservation, and usage accounting.

A Python prototype came first; it lives in the git history at f102fc3 if you want the smaller reference implementation.

License

MIT

Documentation

Overview

codex-prune-proxy — a Responses API proxy that lets the model drop its own spent tool outputs, so they stop being re-sent on every later turn.

The judgment is entirely the model's. The proxy only supplies an inventory of what could be dropped, executes the decisions the model states, and keeps a per-mark ledger. There is no heuristic mode and no extra model call.