compress

package
v1.189.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package compress is ChatCLI's content-aware, reversible context-compression layer. It shrinks the verbose, structured payloads an agent reads — grep/ ripgrep results, build/test logs, unified diffs, large JSON arrays, source code — before they reach the model, while preserving the information the model actually needs to act.

Design goals (in priority order):

  1. Never degrade. The default mode is lossless. Lossy reduction only ever happens when the dropped bytes are first written to a CCR store (Contextual Compression Retrieval) so the model can recover the original verbatim with the @recall tool. Below a size threshold the output is returned byte-identical to the input.
  2. Keyless and self-hosted. Everything here is pure Go — no network, no trained model, no cgo. (Prose/ML compression is a future pluggable backend behind the Compressor interface; it is intentionally absent.)
  3. Content-aware. A ContentRouter detects the payload type (or trusts a source hint such as the originating tool name) and routes to the compressor that understands that structure.

The package is a leaf: it imports only the standard library so it can be used from cli/, cli/plugins/ and the history trimmer without import cycles.

Index

Constants

View Source 
const (
	DefaultThreshold = 4000 // bytes; below this -> verbatim passthrough
	DefaultCCRMaxMB  = 256  // CCR store size cap
	DefaultCCRTTL    = 7 * 24 * time.Hour
)

Default tuning constants. Threshold is intentionally generous: small tool outputs are returned byte-identical, so compression only ever engages on the large payloads where it pays off.

Variables

View Source 
var ErrEntryTooLarge = errors.New("compress: content exceeds the CCR store per-entry capacity")

ErrEntryTooLarge is returned by a bounded Store's Put when the content exceeds the per-entry capacity. Storing it would force the eviction pass to remove the entry itself (along with everything else), leaving any embedded retrieval marker dangling — so the store refuses up front and the caller degrades to passthrough, honoring the never-degrade contract.

Functions

func ExtractKeys 

func ExtractKeys(s string) []string

ExtractKeys returns every distinct CCR key referenced in s, in first-seen order. Used by the @recall tool to resolve markers and by metrics.

func FormatMarker 

func FormatMarker(key string) string

FormatMarker renders the retrieval marker for a key.

func KeyFor 

func KeyFor(content string) string

KeyFor returns the content-addressed CCR key for content. Deterministic and collision-resistant for session-scale volumes.

Types

type CodeCompressor 

type CodeCompressor struct{}

CodeCompressor reduces source code to a structural skeleton — package/imports, type/const/var declarations, and function *signatures* — eliding function bodies. It is the compressor behind Headroom's "codebase exploration" win: when surveying many files the model needs the shape (what exists, what calls what), not every statement. The full source is offloaded to CCR.

SAFETY — this compressor NEVER auto-fires on tool output. Dropping the body of a file an agent is about to edit would be actively harmful (the agent reads it precisely to see/modify the implementation). It therefore only engages when the caller explicitly asks for code compression via Hint.MIME=="code" (e.g. the @compress tool with hint=code, or an explicit codebase-survey path). The automatic tool-output path leaves code untouched.

Go is compressed precisely via go/ast (always valid output). Other languages use a conservative brace/indent heuristic that keeps declaration-shaped lines and drops nested bodies.

func NewCodeCompressor 

func NewCodeCompressor() *CodeCompressor

NewCodeCompressor returns a ready compressor.

func (*CodeCompressor) Compress 

func (c *CodeCompressor) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*CodeCompressor) Detect 

func (c *CodeCompressor) Detect(_ string, h Hint) float64

Detect implements Compressor. It returns confidence ONLY when the caller explicitly flags the content as code; it never sniffs content, so it can never be auto-selected for ordinary tool output.

func (*CodeCompressor) Name 

func (*CodeCompressor) Name() string

Name implements Compressor.

type Compressor 

type Compressor interface {
	// Name is the strategy identifier reported in Result.Strategy.
	Name() string

	// Detect returns the confidence in [0,1] that this compressor is the
	// right one for content, given an optional origin hint. The router picks
	// the highest scorer. A score of 0 means "not my content".
	Detect(content string, h Hint) float64

	// Compress reduces content under opts. It must honor opts.Mode and the
	// reversibility contract: in ModeLossyWithCCR with a nil Store it must
	// not drop information. Returning Reversible=false is a programming
	// error; the router will reject it and fall back to passthrough.
	Compress(content string, opts Options) (Result, error)
}

Compressor reduces one class of content. Implementations must be safe for concurrent use — the router may invoke them from multiple goroutines.

type Config 

type Config struct {
	Mode      Mode
	Profile   Profile
	Threshold int
	Store     Store
}

Config controls Layer construction. Fields left zero take documented defaults (see NewLayerFromEnv).

type ContentRouter 

type ContentRouter struct {
	// contains filtered or unexported fields
}

ContentRouter is the entry point of the compression layer. It holds an ordered set of Compressors, detects (or is told via a Hint) which one fits a payload, runs it, and enforces the package-wide safety contract:

  • ModeOff or below Threshold -> verbatim passthrough.
  • The chosen compressor must return Reversible=true. If it ever returns an irreversible Result (a bug), the router discards it and falls back to passthrough rather than silently degrading the model's context.
  • A "reduction" that grew the payload (CompressedSize > OriginalSize) is discarded in favor of the original — compression never makes prompts bigger.

The router is safe for concurrent use as long as its Compressors are.

func NewContentRouter 

func NewContentRouter(compressors ...Compressor) *ContentRouter

NewContentRouter builds a router over the given compressors, tried in descending Detect-confidence order. Pass them in any order; selection is by score, not position.

func (*ContentRouter) Compress 

func (r *ContentRouter) Compress(content string, h Hint, opts Options) Result

Compress reduces content according to opts, routing to the best-matching compressor. It always returns a usable Result — never an error to the caller's hot path — because a compression failure must degrade to passthrough, not break the agent turn. (Compressor errors are reflected by returning the passthrough Result.)

type DiffCompressor 

type DiffCompressor struct {
	MaxContextLines int
	MaxHunksPerFile int
	MaxFiles        int
}

DiffCompressor reduces unified-diff output (git diff / git show). The information that matters is the changed lines (+/-) and the hunk headers; long runs of unchanged context lines are noise the model rarely needs. This compressor keeps every addition and deletion, trims context to a small window around each change, and caps hunks-per-file and files. Dropped context is offloaded to CCR.

func NewDiffCompressor 

func NewDiffCompressor() *DiffCompressor

NewDiffCompressor returns a compressor with Headroom-equivalent caps.

func NewDiffCompressorFor added in v1.150.0 

func NewDiffCompressorFor(p Profile) *DiffCompressor

NewDiffCompressorFor returns a compressor tuned for the given profile: conservative keeps roughly double the default caps, aggressive roughly half.

func (*DiffCompressor) Compress 

func (c *DiffCompressor) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*DiffCompressor) Detect 

func (c *DiffCompressor) Detect(content string, h Hint) float64

Detect implements Compressor.

func (*DiffCompressor) Name 

func (*DiffCompressor) Name() string

Name implements Compressor.

type DiskStore 

type DiskStore struct {
	// contains filtered or unexported fields
}

DiskStore is a bounded, content-addressed, crash-safe on-disk Store.

Each original is written to "<dir>/<key>.ccr" as raw bytes. Because the filename *is* the content hash, the store needs no separate index file that could be corrupted or drift from reality — the directory is the index. File modification time doubles as the last-access timestamp (refreshed on Put and Get), which drives both TTL pruning and LRU eviction when the total size exceeds the cap.

func NewDiskStore 

func NewDiskStore(dir string, maxBytes int64, ttl time.Duration) (*DiskStore, error)

NewDiskStore opens (creating if needed) a bounded store rooted at dir. A maxBytes <= 0 disables the size cap; a ttl <= 0 disables TTL pruning. On open it scans existing entries, prunes any past their TTL, and evicts down to the cap so a restart inherits a healthy footprint.

A bounded store also enforces a per-entry capacity (maxBytes/4): Put returns ErrEntryTooLarge for content that would immediately fall out of the cap, instead of accepting it and letting eviction leave the returned key dangling.

func (*DiskStore) Get 

func (s *DiskStore) Get(key string) (string, bool, error)

Get implements Store. A hit refreshes the entry's recency.

func (*DiskStore) Prune added in v1.147.0 

func (s *DiskStore) Prune() PruneResult

Prune implements Store: a directory rescan (reconciling with other processes), TTL prune, and size-cap eviction, run on demand. The before/after delta is computed over the reconciled view so the report reflects what this pass actually freed, not stale accounting.

func (*DiskStore) Put 

func (s *DiskStore) Put(content string) (string, error)

Put implements Store. The write is atomic (temp file + rename) so a crash never leaves a partial original under a valid content hash.

func (*DiskStore) Stats 

func (s *DiskStore) Stats() StoreStats

Stats implements Store. Beyond the raw footprint it computes the least-recently-accessed entry's age and how many entries are already past the TTL, so the /config surface can show curation status.

type Hint 

type Hint struct {
	// ToolName is the originating tool, e.g. "@search", "@read", "git diff".
	// The router maps known tools straight to a compressor.
	ToolName string

	// Filename is the path the content came from, when known. Its extension
	// helps the code compressor pick a language.
	Filename string

	// MIME is an explicit content type when the caller already knows it.
	MIME string
}

Hint carries out-of-band signals about a payload's origin so the router can route with high confidence instead of guessing from content alone. All fields are optional.

type JSONCrusher 

type JSONCrusher struct {
	MinItemsToCrush int
	HeadItems       int
	TailItems       int
}

JSONCrusher reduces JSON payloads — the API responses, config dumps and tabular tool outputs an agent reads. It has two modes, applied in order:

  • Lossless: re-canonicalize pretty-printed JSON, eliding insignificant whitespace (json.Compact). Always reversible, never needs CCR.
  • Lossy (arrays only): for a large top-level array, keep a representative head and tail of elements and replace the dropped middle with a single "_ccr_dropped" sentinel element carrying the @recall marker. The output is still valid JSON; the full array is offloaded to CCR.

The lossy path mirrors Headroom's SmartCrusher sentinel ({"_ccr_dropped":"<<ccr:HASH ...>>"}) so downstream consumers can skip the sentinel with isCCRSentinel.

func NewJSONCrusher 

func NewJSONCrusher() *JSONCrusher

NewJSONCrusher returns a crusher with sensible defaults.

func NewJSONCrusherFor added in v1.150.0 

func NewJSONCrusherFor(p Profile) *JSONCrusher

NewJSONCrusherFor returns a crusher tuned for the given profile: conservative keeps roughly double the default sample, aggressive roughly half.

func (*JSONCrusher) Compress 

func (c *JSONCrusher) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*JSONCrusher) Detect 

func (c *JSONCrusher) Detect(content string, h Hint) float64

Detect implements Compressor.

func (*JSONCrusher) Name 

func (*JSONCrusher) Name() string

Name implements Compressor.

type Layer 

type Layer struct {
	// contains filtered or unexported fields
}

Layer is the high-level facade the rest of ChatCLI talks to. It bundles a ContentRouter over every built-in compressor, a CCR store, the active mode/ threshold, and a metrics accumulator. One Layer is created per session and shared (it is safe for concurrent use).

The zero Layer is not usable; build one with NewLayer or NewLayerFromEnv.

func NewLayer 

func NewLayer(cfg Config) *Layer

NewLayer builds a Layer from an explicit Config. A nil Store with ModeLossyWithCCR is allowed: lossy compressors degrade to lossless, never dropping information.

func NewLayerFromEnv 

func NewLayerFromEnv(stateDir string) *Layer

NewLayerFromEnv builds a Layer from environment configuration, creating the on-disk CCR store under stateDir/ccr. stateDir is typically ~/.chatcli; when empty it is resolved from the user home (falling back to the temp dir).

Recognized variables: 

CHATCLI_COMPRESSION            off | lossless | lossy-with-ccr (default lossy-with-ccr)
CHATCLI_COMPRESSION_PROFILE    conservative | default | aggressive (default default)
CHATCLI_COMPRESSION_THRESHOLD  bytes below which output is untouched (default 4000)
CHATCLI_COMPRESSION_CCR_DIR    override the CCR store directory
CHATCLI_COMPRESSION_CCR_MAX_MB CCR size cap in MiB (default 256; 0 = unbounded)
CHATCLI_COMPRESSION_CCR_TTL    CCR entry TTL as a Go duration (default 168h; 0 = no TTL)

The CCR knobs nest under the CHATCLI_COMPRESSION_ prefix to match the subsystem-prefix convention used across ChatCLI (CHATCLI_AGENT_*, CHATCLI_QUALITY_*, CHATCLI_MICROCOMPACT_*, ...).

func (*Layer) Archive added in v1.155.0 

func (l *Layer) Archive(content string) (string, bool)

Archive stores content in the CCR store verbatim and returns its retrieval key, bypassing the compression router entirely. Compaction paths that build their own stub text (microcompact previews, emergency history shrinking) use this so the bytes they drop from the conversation stay recoverable through @recall. Returns ok=false when the layer is disabled, the content already carries a CCR marker (its original is archived under that key — a second copy would waste store capacity), or the store rejects the entry (e.g. over the per-entry cap).

func (*Layer) CompressHinted 

func (l *Layer) CompressHinted(h Hint, content string) (string, Result)

CompressHinted reduces content using an explicit routing hint. This is the entry point for callers that know the content type out of band — e.g. the @compress tool passing Hint.MIME=="code" to request code skeletonization, which never happens on the automatic path. A nil or disabled Layer returns the input unchanged.

func (*Layer) CompressToolOutput 

func (l *Layer) CompressToolOutput(toolName, content string) (string, Result)

CompressToolOutput reduces one tool's output, attributing the result to the originating tool for routing and metrics. It always returns a usable string; a nil or disabled Layer returns the input unchanged.

func (*Layer) Enabled 

func (l *Layer) Enabled() bool

Enabled reports whether compression will engage (mode is not off).

func (*Layer) Mode 

func (l *Layer) Mode() Mode

Mode reports the Layer's active mode. Safe for concurrent use.

func (*Layer) Profile added in v1.150.0 

func (l *Layer) Profile() Profile

Profile reports the Layer's active aggressiveness profile. Safe for concurrent use.

func (*Layer) Prune added in v1.147.0 

func (l *Layer) Prune() PruneResult

Prune curates the CCR store now (drop TTL-expired entries, evict to the size cap) and returns what was removed. Used by `/config compression prune`. A nil layer/store, or a Store that does not implement Pruner, is a no-op returning a zero result.

func (*Layer) Recall 

func (l *Layer) Recall(key string) (string, bool)

Recall returns the original content stored under a CCR key, or ok=false when the key is unknown/evicted. Used by the @recall tool.

func (*Layer) SetMode 

func (l *Layer) SetMode(m Mode)

SetMode changes the active mode at runtime (used by /config compression). Safe for concurrent use; takes effect on the next CompressToolOutput call.

func (*Layer) SetProfile added in v1.150.0 

func (l *Layer) SetProfile(p Profile)

SetProfile changes the aggressiveness profile at runtime (used by /config compression profile). The router is rebuilt with the new caps and swapped atomically, so in-flight Compress calls finish on the old tuning and subsequent calls pick up the new one. Safe for concurrent use.

func (*Layer) SetThreshold added in v1.150.0 

func (l *Layer) SetThreshold(n int)

SetThreshold changes the engage threshold at runtime (used by /config compression threshold). Values <= 0 mean "always attempt compression". Safe for concurrent use.

func (*Layer) Stats 

func (l *Layer) Stats() (Stats, StoreStats)

Stats returns a snapshot of compression metrics plus the CCR store footprint, for /compression stats and the cost footer.

func (*Layer) StoreFallback added in v1.150.0 

func (l *Layer) StoreFallback() error

StoreFallback reports why the persistent CCR store could not be opened, or nil when the configured store is active. A non-nil value means the layer is running on a bounded in-memory store: compression still works and markers still recall within this process, but offloaded originals do not survive a restart and are not shared with other ChatCLI processes.

func (*Layer) Threshold added in v1.150.0 

func (l *Layer) Threshold() int

Threshold reports the minimum payload size (bytes) that engages compression. Safe for concurrent use.

type LogCompressor 

type LogCompressor struct {
	MaxErrors         int
	ErrorContextLines int
	MaxStackTraces    int
	StackTraceMaxLine int
	MaxWarnings       int
	MaxTotalLines     int
}

LogCompressor reduces build/test/CI/runtime log output — the payload behind Headroom's "SRE incident debugging 65,694 -> 5,118 tokens (92%)". Logs are mostly low-signal INFO/DEBUG noise punctuated by a few high-signal events: errors, stack traces, deduplicated warnings, and the final summary. This compressor keeps the signal and offloads the noise to CCR.

Two correctness behaviors ported from the reference Rust port:

  • Stack traces survive blank lines. A Python traceback often contains a blank line mid-trace; a naive "stop at blank line" state machine would truncate it. We keep contiguous frame runs across single blanks.
  • Warning dedupe is conservative. We split each warning on its first ':' or '=' and dedupe on the (lower-cased) head only. Warnings sharing a category head collapse to one representative, while distinct categories (different heads) are always kept — we never merge unrelated warnings.

func NewLogCompressor 

func NewLogCompressor() *LogCompressor

NewLogCompressor returns a compressor with Headroom-equivalent caps.

func NewLogCompressorFor added in v1.150.0 

func NewLogCompressorFor(p Profile) *LogCompressor

NewLogCompressorFor returns a compressor tuned for the given profile: conservative keeps roughly double the default caps, aggressive roughly half.

func (*LogCompressor) Compress 

func (c *LogCompressor) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*LogCompressor) Detect 

func (c *LogCompressor) Detect(content string, h Hint) float64

Detect implements Compressor. Logs are recognized by a mix of level tokens, stack-trace frames, and test-runner summaries over many lines.

func (*LogCompressor) Name 

func (*LogCompressor) Name() string

Name implements Compressor.

type MemoryStore 

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is an in-process Store. Unbounded by default (tests and the one-shot -p path, where sessions are short and nothing should touch disk); NewBoundedMemoryStore adds the same LRU size cap and per-entry capacity as DiskStore, for use as the long-running fallback when the disk store cannot be opened. TTL is deliberately absent: it exists to curate entries across restarts, and a memory store never survives one — the LRU cap is what bounds a long-lived process.

func NewBoundedMemoryStore added in v1.150.0 

func NewBoundedMemoryStore(maxBytes int64) *MemoryStore

NewBoundedMemoryStore returns an in-memory store bounded at maxBytes with LRU eviction and a per-entry capacity of maxBytes/4 (see ErrEntryTooLarge). A maxBytes <= 0 yields an unbounded store, same as NewMemoryStore.

func NewMemoryStore 

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty, unbounded in-memory store.

func (*MemoryStore) Get 

func (m *MemoryStore) Get(key string) (string, bool, error)

Get implements Store. A hit refreshes the entry's recency.

func (*MemoryStore) Prune added in v1.147.0 

func (m *MemoryStore) Prune() PruneResult

Prune implements Pruner: evicts down to the size cap (a no-op when unbounded, since Put keeps a bounded store within its cap continuously).

func (*MemoryStore) Put 

func (m *MemoryStore) Put(content string) (string, error)

Put implements Store.

func (*MemoryStore) Stats 

func (m *MemoryStore) Stats() StoreStats

Stats implements Store.

type Metrics 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics accumulates compression accounting for a session. It mirrors the counter-with-snapshot pattern used elsewhere (e.g. cache_planner's cacheBlocksCoalesced) and is safe for concurrent use. A nil *Metrics is a valid no-op receiver, so callers never need a nil check.

func NewMetrics 

func NewMetrics() *Metrics

NewMetrics returns a ready accumulator.

func (*Metrics) RecordCCRDedupe 

func (m *Metrics) RecordCCRDedupe()

func (*Metrics) RecordCCRHit 

func (m *Metrics) RecordCCRHit()

func (*Metrics) RecordCCRMiss 

func (m *Metrics) RecordCCRMiss()

func (*Metrics) RecordCCRPut 

func (m *Metrics) RecordCCRPut()

RecordCCRPut / RecordCCRDedupe / RecordCCRHit / RecordCCRMiss track the reversible-store side of the layer. All safe on a nil receiver.

func (*Metrics) RecordCompression 

func (m *Metrics) RecordCompression(r Result)

RecordCompression accounts for one Compress call. Safe on a nil receiver.

func (*Metrics) Snapshot 

func (m *Metrics) Snapshot() Stats

Snapshot returns a consistent copy of the current counters.

type Mode 

type Mode int32

Mode selects how aggressively the layer reduces a payload. Backed by int32 so it stores directly into the atomic on Layer without a widening conversion. 

const (
	// ModeOff disables compression entirely; Compress is a verbatim
	// passthrough. Used when the user sets CHATCLI_COMPRESSION=off.
	ModeOff Mode = iota

	// ModeLosslessOnly applies only reductions that lose no information
	// (e.g. JSON re-canonicalization, whitespace normalization). The output
	// is always fully reconstructable without a CCR lookup.
	ModeLosslessOnly

	// ModeLossyWithCCR additionally allows dropping low-value rows/lines/
	// hunks, but only after the original payload is persisted to the CCR
	// store and a retrieval marker is embedded in the output. Nothing is
	// truly lost — the model can @recall the original.
	ModeLossyWithCCR
)

func ParseMode 

func ParseMode(s string) (Mode, bool)

ParseMode maps a config/env string to a Mode. Unknown values fall back to ModeLossyWithCCR (the recommended default) and report ok=false so callers can warn.

func (Mode) String 

func (m Mode) String() string

String renders the mode for /config and logs.

type Options 

type Options struct {
	// Mode selects the reduction strategy. See the Mode constants.
	Mode Mode

	// Store is the CCR backend used to offload dropped originals in
	// ModeLossyWithCCR. When nil, lossy compressors degrade to their
	// lossless behavior (no row/line dropping) so reversibility is never
	// violated.
	Store Store

	// Threshold is the minimum input size, in bytes, below which Compress is
	// a verbatim passthrough regardless of Mode. Guarantees small payloads
	// are byte-identical to today's behavior. A value <= 0 means "always
	// attempt compression".
	Threshold int

	// Metrics, when non-nil, receives per-call accounting. Safe for
	// concurrent use.
	Metrics *Metrics
}

Options configures a single Compress call. The zero value is safe: it behaves as ModeOff with no store (verbatim passthrough).

type Profile added in v1.150.0 

type Profile int32

Profile selects how much content the lossy compressors keep before offloading the rest to CCR. It tunes the caps, never the safety contract: every profile remains fully reversible via @recall, and lossless-only or off modes ignore the profile entirely.

Backed by int32 so it stores directly into the atomic on Layer. 

const (
	// ProfileDefault is the Headroom-equivalent tuning the ratios eval was
	// calibrated against. The recommended setting.
	ProfileDefault Profile = iota

	// ProfileConservative keeps roughly twice as much context per payload.
	// For users who prefer fewer @recall round-trips over maximum savings.
	ProfileConservative

	// ProfileAggressive keeps roughly half as much context per payload.
	// For long unattended agent runs where context headroom is the priority.
	ProfileAggressive
)

func ParseProfile added in v1.150.0 

func ParseProfile(s string) (Profile, bool)

ParseProfile maps a config/env string to a Profile. Unknown values fall back to ProfileDefault and report ok=false so callers can warn.

func (Profile) String added in v1.150.0 

func (p Profile) String() string

String renders the profile for /config and logs.

type ProseCompressor 

type ProseCompressor struct {
	// SectionCap is the max chars kept for a single section before head/tail
	// trimming kicks in. Sections shorter than this are kept whole.
	SectionCap int
}

ProseCompressor reduces prose / Markdown — the HTML-turned-Markdown an agent gets back from web fetches and searches. That content is dominated by boilerplate: navigation menus, cookie banners, "skip to content", footers repeated across every page, plus long runs of blank lines. This compressor strips the repetition keylessly (no ML model, unlike Headroom's Kompress) and offloads the full original to CCR.

It is deliberately conservative — it removes only *exact duplicate* non-empty lines (keeping the first occurrence) and collapses blank-line runs, then trims only individual sections that are extraordinarily long. Unique prose is never dropped.

SCOPE — to honor "never degrade", prose compression auto-fires ONLY on reference material fetched from the web (Detect keys off the web tool hints), or when explicitly requested via Hint.MIME=="prose"/"markdown". It never engages on local file reads (@read), which an agent may be about to edit.

func NewProseCompressor 

func NewProseCompressor() *ProseCompressor

NewProseCompressor returns a compressor with sensible defaults.

func NewProseCompressorFor added in v1.150.0 

func NewProseCompressorFor(p Profile) *ProseCompressor

NewProseCompressorFor returns a compressor tuned for the given profile: conservative keeps roughly double the default section budget, aggressive roughly half.

func (*ProseCompressor) Compress 

func (c *ProseCompressor) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*ProseCompressor) Detect 

func (c *ProseCompressor) Detect(_ string, h Hint) float64

Detect implements Compressor. It engages on web/reference tool output and on an explicit prose/markdown hint — never by sniffing arbitrary text, so it can't silently rewrite content an agent is editing.

func (*ProseCompressor) Name 

func (*ProseCompressor) Name() string

Name implements Compressor.

type PruneResult added in v1.147.0 

type PruneResult struct {
	Removed          int
	BytesFreed       int64
	RemainingEntries int
	RemainingBytes   int64
}

PruneResult reports what a curation pass removed and what remains, so the user gets concrete feedback ("freed N entries / X bytes") instead of a silent cleanup.

type Pruner added in v1.147.0 

type Pruner interface {
	// Prune curates the store now and returns what was removed. Idempotent
	// and safe to call at any time.
	Prune() PruneResult
}

Pruner is the optional capability of a Store that curates itself on demand — dropping TTL-expired entries and evicting down to the size cap. It is kept separate from Store (probed via a type assertion in Layer.Prune) so adding curation does not break the Store contract for existing implementations.

type Result 

type Result struct {
	// Compressed is the reduced payload to send to the model. When no
	// reduction was applied it equals the input.
	Compressed string

	// OriginalSize and CompressedSize are byte lengths, for ratio reporting.
	OriginalSize   int
	CompressedSize int

	// Strategy names the compressor that handled the payload, e.g. "search",
	// "log", "diff", "json-crush", "code-ast", or "passthrough".
	Strategy string

	// CacheKey is the CCR key under which the full original was stored, or ""
	// when nothing was offloaded (lossless or passthrough). When set, the
	// Compressed payload contains a retrieval marker (see FormatMarker).
	CacheKey string

	// Reversible is true when the original is fully recoverable — either
	// because the reduction was lossless, or because the dropped bytes were
	// offloaded to CCR. The layer never returns an irreversible Result.
	Reversible bool

	// Detail carries compressor-specific counters (e.g. "matches_kept",
	// "lines_dropped") for diagnostics. May be nil.
	Detail map[string]int
}

Result is the outcome of compressing one payload.

func (Result) Ratio 

func (r Result) Ratio() float64

Ratio is CompressedSize/OriginalSize in [0,1]; 1.0 means no reduction.

func (Result) SavedBytes 

func (r Result) SavedBytes() int

SavedBytes reports how many bytes the reduction removed from the prompt (never negative).

type SearchCompressor 

type SearchCompressor struct {
	MaxMatchesPerFile int
	MaxTotalMatches   int
	MaxFiles          int
}

SearchCompressor reduces grep/ripgrep output. Code-search results are the single highest-volume, highest-redundancy payload an agent reads: hundreds of "path:line:content" rows where the model only needs a representative sample per file plus anything that looks like an error. This is the compressor behind Headroom's headline "100 results 17,765 -> 1,408 tokens".

Parser robustness (the bugs the reference Rust port fixed, ported here):

  • Windows drive letters: "C:\src\main.go:42:hit" must not treat the drive colon as the line-number separator.
  • Dashes in filenames: ripgrep context lines use "path-42-content"; a path like "pre-commit-config.yaml" must still parse. We anchor on the earliest "<sep>\d+<sep>" marker rather than a fixed character class.

func NewSearchCompressor 

func NewSearchCompressor() *SearchCompressor

NewSearchCompressor returns a compressor with Headroom-equivalent caps.

func NewSearchCompressorFor added in v1.150.0 

func NewSearchCompressorFor(p Profile) *SearchCompressor

NewSearchCompressorFor returns a compressor tuned for the given profile: conservative keeps roughly double the default caps, aggressive roughly half.

func (*SearchCompressor) Compress 

func (c *SearchCompressor) Compress(content string, opts Options) (Result, error)

Compress implements Compressor.

func (*SearchCompressor) Detect 

func (c *SearchCompressor) Detect(content string, h Hint) float64

Detect implements Compressor. Confidence rises with the fraction of lines that parse as "path:line:content" rows.

func (*SearchCompressor) Name 

func (*SearchCompressor) Name() string

Name implements Compressor.

type Stats 

type Stats struct {
	Calls      int64
	Reductions int64
	BytesIn    int64
	BytesOut   int64
	CCRPuts    int64
	CCRDedupes int64
	CCRHits    int64
	CCRMisses  int64
	ByStrategy []StrategyStat
}

Stats is an immutable snapshot of the accumulated metrics, suitable for /compression stats and the cost footer.

func (Stats) Ratio 

func (s Stats) Ratio() float64

Ratio is the aggregate BytesOut/BytesIn in [0,1]; 1.0 means no reduction.

func (Stats) RecallHitRate added in v1.150.0 

func (s Stats) RecallHitRate() (int, bool)

RecallHitRate returns the percentage (0–100) of @recall lookups that were served, and ok=false when no lookups have happened yet. A persistently low rate signals over-aggressive eviction (cap too small / TTL too short) — markers are being promised that the store can no longer honor.

func (Stats) SavedBytes 

func (s Stats) SavedBytes() int64

SavedBytes is the total prompt reduction (never negative).

type Store 

type Store interface {
	// Put stores content and returns its content-addressed key. Storing the
	// same content again is idempotent (same key, no duplicate write) and
	// refreshes the entry's recency for eviction purposes.
	Put(content string) (key string, err error)

	// Get returns the original for key. ok is false when the key is unknown
	// or has been evicted.
	Get(key string) (content string, ok bool, err error)

	// Stats reports the current footprint.
	Stats() StoreStats
}

Store persists compression originals for on-demand retrieval. Implementations must be safe for concurrent use.

type StoreStats 

type StoreStats struct {
	Entries    int
	TotalBytes int64
	MaxBytes   int64
	// Curation visibility: age of the least-recently-accessed entry, how many
	// entries are already past the TTL (i.e. would be removed by a prune), and
	// the configured TTL (0 = disabled). Let the /config surface show that
	// curation is happening rather than leaving the store opaque.
	OldestAge    time.Duration
	StaleEntries int
	TTL          time.Duration
}

StoreStats is a point-in-time snapshot of a Store's footprint.

type StrategyStat 

type StrategyStat struct {
	Strategy string
	Calls    int64
	BytesIn  int64
	BytesOut int64
}

StrategyStat is a per-strategy line in a Snapshot.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.