compress

Documentation

Overview

Package compress is the content-aware tool-output compressor library: a gate decides whether content is safe to compress, a router classifies it, and a pipeline dispatches to specialized compressors. Everything on this path runs on every tool call, so all regexes are precompiled package vars and the hot path avoids per-call allocation where practical.

Index

• Constants
• func CRSegmentsLookLikeRewrites(segs []string) bool
• func Decide(content string, nTokens int, toolName, mime, contentClass string, ...) (action, klass, signal, reason string)
• func EstimateTokens(s string) int
• func SegmentMarkdown(lines []string) ([]MDSegment, MDStats)
• func StripANSI(s string) string
• func StripLineNumbers(content string) string
• type Compressor
• type ContentType
  • func Detect(content string) (ContentType, float64)
• type GateConfig
  • func DefaultGateConfig() GateConfig
• type Input
• type Logger
• type MDSegment
• type MDStats
• type Outcome
• type Pipeline
  • func NewPipeline(r *Registry, gate GateConfig, log Logger) *Pipeline
  • func (p *Pipeline) Compress(ctx context.Context, in Input) (outcome Outcome)
• type Registry
  • func NewRegistry(chains map[ContentType][]Compressor) *Registry
  • func (r *Registry) Chain(ct ContentType) []Compressor
• type Result

Constants

const (
	DefaultMinTokens     = 64
	DefaultMaxChars      = 262144 // 256 KiB (~64k tokens): global, deterministic-safe
	DefaultProseMaxChars = 4500   // LLMLingua-only ceiling: what fits the 2s hook budget

)

Gate defaults. There are two distinct ceilings, and conflating them was wrong:

• DefaultMaxChars is a GLOBAL safety bound: content larger than this is skipped before classification, purely to cap classify/regex cost on a pathological input. Deterministic structural compressors (Log/JSON/ANSI) crush large tool output cheaply, so this is generous, not the prose limit.
• DefaultProseMaxChars is a LATENCY-BUDGET ceiling for the prose path, NOT a model-context one. Measured end-to-end (prod, llmlingua-2 xlm-roberta-large on 4 vCPU, single-flight): latency ≈ 0.3s + 0.3s/KB, so 4.5KB ≈ 1.5s, 6KB ≈ 2.1s (breaches the edge hook's 2s budget), ≥9KB always times out. The old 30000 value admitted 5x more than the budget could serve — every 6-30KB prose burned the full 2s and surfaced as an error. Above this ceiling prose skips CLEANLY upfront. Raise only after a faster model (bert-base ≈ 3x throughput) or chunked-parallel inference lands.

Functions

func CRSegmentsLookLikeRewrites

func CRSegmentsLookLikeRewrites(segs []string) bool

CRSegmentsLookLikeRewrites reports whether a CR-chain's segments are frames re-rendering the same content (terminal overwrite: collapse-safe) rather than unrelated records (`\r`-delimited data: collapse would be total loss). Frames share a long common prefix ("Downloading 4%" / "Downloading 5%") or suffix ("⠋ building" / "⠙ building"); data records share neither. Single source of truth for both the router's lone-CR terminal signal and the ANSIStrip CR-overwrite collapse (the StripANSI pattern).

Requires >=2/3 of adjacent non-empty pairs to be similar; <2 non-empty segments trivially qualify (nothing to collapse).

func Decide

func Decide(content string, nTokens int, toolName, mime, contentClass string, minTokens int) (action, klass, signal, reason string)

Decide is the gate verdict, ported from gate.py decide(). action ∈ {compress, skip}. too_short takes precedence over code_structured, matching gate.py exactly.

func EstimateTokens

func EstimateTokens(s string) int

EstimateTokens is a fast, dependency-free token-count estimator calibrated against tiktoken o200k_base (docs/compressor-opportunities.md #3). The consumer of compressed output pays TOKENS, not bytes, and bytes diverge from tokens by up to ~4x on whitespace-heavy content — so encoding choices and accept gates use this, not len().

Single pass; integer ops only. Calibrated by grid-search on 10 content classes (prose, minified JSON, CSV, Go code, logs, padded tables, markdown, unicode, numbers, YAML) against real tiktoken: MAE 8.2%, worst class +16.3% (go code), vs len/4's systematic -27..-48% on structured content. The bias is a slight consistent OVERestimate, which is the safe direction for an accept gate, and cancels when comparing candidate encodings of the same content.

Model: ASCII letter run of length L = 1 + (L-1)/8 tokens; digit run = ceil(L/3); mixed punctuation/symbol run = ceil(L/4) (BPE merges glue like "," ":" "},{"); space/tab runs of >1 = 1 + (L-2)/6 (single spaces merge into the next word); newline runs = 1; every non-ASCII rune = 1.

func SegmentMarkdown

func SegmentMarkdown(lines []string) ([]MDSegment, MDStats)

SegmentMarkdown classifies lines into verbatim/prose segments and gathers the gate's evidence in one pass. Line-oriented CommonMark-lite; every rule that could go either way goes VERBATIM:

• fenced code: ``` / ~~~ open (<=3 leading spaces, any info string) through a closing fence of the same char and >= length. An UNCLOSED fence makes the REST OF THE DOCUMENT verbatim (truncated docs must not leak code to the model). Everything inside is verbatim and exempt from veto counting.
• indented code (>=4 spaces or tab before non-space).
• ATX headings; setext headings (the underlined TEXT line is verbatim too); thematic breaks; tables (>=2 '|'); blockquotes; list items; link-reference definitions; HTML-leading and image-leading lines.
• config/script stragglers outside fences (kv/assign/docker/code-signal lines): verbatim AND counted in VetoLines — sparse ones are normal in real docs ("Magic: 0xE85250D6"), dominance means the file is config, not a doc (the caller applies the budget).
• shebang: flagged; callers treat it as an absolute veto (scripts).
• blank lines: attached to the current prose run (paragraph separators the model force-keeps); a blank inside a verbatim run splits it, which only costs an extra sentinel.

func StripANSI

func StripANSI(s string) string

StripANSI removes terminal escape sequences (CSI, OSC, DCS, charset, single-char) and is the single source of truth for both detection and the ANSIStrip compressor. Lossless on visible text. Returns s unchanged when it has no escapes.

func StripLineNumbers

func StripLineNumbers(content string) string

StripLineNumbers removes `N\t` line-number prefixes from a whole document. Single source of truth with detectLineNumbered's regex (the StripANSI pattern), used by the doc_read chain's LineNumberStrip compressor.

Types

type Compressor

type Compressor interface {
	Name() string
	Handles(ContentType) bool
	Compress(ctx context.Context, in Input) (Result, error)
}

Compressor transforms content of the types it Handles. Implementations must be safe for concurrent use (the pipeline holds one instance per type).

type ContentType

type ContentType string

ContentType is the router's classification of a piece of content.

const (
	TypeJSON    ContentType = "json"
	TypeCode    ContentType = "code"
	TypeLog     ContentType = "log"
	TypeDiff    ContentType = "diff"
	TypeHTML    ContentType = "html"
	TypeSearch  ContentType = "search"
	TypeTabular ContentType = "tabular"
	// TypeDocRead is a line-numbered file read (`N\t<line>`, Read tool / cat -n)
	// whose STRIPPED content is unmistakably a markdown/prose document
	// (isMarkdownDoc). Routed to the prose model after LineNumberStrip; every
	// other line-numbered read stays TypeCode (passthrough) — code must never
	// reach the paraphraser.
	TypeDocRead  ContentType = "doc_read"
	TypeProse    ContentType = "prose"
	TypeTerminal ContentType = "terminal"
	TypeUnknown  ContentType = "unknown"
)

func Detect

func Detect(content string) (ContentType, float64)

Detect classifies content. Ordered, first-match-wins: each detector returns a confidence and is taken only if it clears its gate. Falls back to prose.

The content is split into lines ONCE (capped) and that slice is shared by every line-based detector — re-splitting per detector was the hot-path cost.

type GateConfig

type GateConfig struct {
	MinTokens int
	MaxChars  int // global ceiling: skip before classify; deterministic compressors handle the rest
	// ProseMaxChars caps ONLY the prose path (LLMLingua). 0 disables the cap.
	ProseMaxChars int
}

GateConfig bounds what the gate will accept.

func DefaultGateConfig

func DefaultGateConfig() GateConfig

DefaultGateConfig returns the ported gate.py / app.py defaults.

type Input

type Input struct {
	Content      string
	ContentType  ContentType
	ToolName     string
	MIME         string
	ContentClass string
	Rate         float64
	MinTokens    int
}

Input is the unit of work handed to the pipeline and each compressor.

ContentType is an OPTIONAL routing override: when empty the router detects the type from Content. MIME and ContentClass carry the raw request gating signals that gate.py consumes (HTTP content_type MIME hint and the explicit prose|code_structured override); they are not part of the Headroom Input but are required to port the gate faithfully.

type Logger

type Logger interface {
	Printf(format string, args ...any)
}

Logger is the minimal logging surface the pipeline needs (satisfied by the stdlib *log.Logger). Kept tiny so callers aren't forced into a logging dep.

type MDSegment

type MDSegment struct {
	Verbatim bool
	Text     string // original lines joined with "\n", byte-exact
}

MDSegment is one contiguous run of lines of a single kind.

type MDStats

type MDStats struct {
	Headings   int // ATX/setext headings (verbatim, but counted as doc evidence)
	Sentences  int // prose lines that read as sentences (>=5 words ending '.')
	ProseChars int // total bytes classified prose
	TotalLines int // non-blank lines
	VetoLines  int // config/script-signature lines OUTSIDE fences (kv/assign/docker/code)
	Shebang    bool
}

MDStats summarizes a segmentation for the routing gate.

type Outcome

type Outcome struct {
	Output     string
	Action     string // "compressed" | "skipped"
	SkipReason string // "" when compressed
	Strategy   string // "+"-joined compressor names that ran
	Detected   ContentType
	GateKlass  string
	GateSignal string
	InChars    int
	OutChars   int
}

Outcome is the full result of running the pipeline once.

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline ties the gate, router and registry together. It is safe for concurrent use: it holds no mutable state.

func NewPipeline

func NewPipeline(r *Registry, gate GateConfig, log Logger) *Pipeline

NewPipeline wires a pipeline. log may be nil.

func (*Pipeline) Compress

func (p *Pipeline) Compress(ctx context.Context, in Input) (outcome Outcome)

Compress runs gate → route → chain → guardrail. It NEVER returns an error: every failure is fail-open (the original content passes through, action "skipped"). The contract mirrors the Python service so the edge-server caller — which reads only Output(compressed) + Action — is unchanged.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry maps a ContentType to its ordered, chainable compressors. It is a pure data structure: the concrete chains are wired by the caller (see the compressors package's DefaultChains) so this package never imports the concrete compressors — that would be an import cycle.

func NewRegistry

func NewRegistry(chains map[ContentType][]Compressor) *Registry

NewRegistry builds a Registry from a type→chain map. The map is used as-is; callers should not mutate it after construction (the pipeline reads it concurrently without locking).

func (*Registry) Chain

func (r *Registry) Chain(ct ContentType) []Compressor

Chain returns the ordered compressors for a content type, or nil if none are registered (the pipeline treats nil as "no compressor available").

type Result

type Result struct {
	Output   string
	Strategy string
	InChars  int
	OutChars int
	// Skipped is a clean, non-error skip (the compressor chose not to compress,
	// e.g. an upstream sidecar shed load or gated the input). The pipeline treats
	// it as a passthrough skip with SkipReason, NOT as the error fail-open path —
	// so legitimate skips do not pollute the error rate.
	Skipped    bool
	SkipReason string
}

Result is what a single compressor returns.