Documentation
¶
Overview ¶
Package compressors holds the concrete Compressor implementations and wires the default per-type chains. It imports the compress package for the core types; the compress package never imports back (no cycle).
Index ¶
- Constants
- func ChainsWithModel(modelURL string) map[compress.ContentType][]compress.Compressor
- func DefaultChains() map[compress.ContentType][]compress.Compressor
- type ANSIStrip
- type JSONCrusher
- type LLMLinguaAdapter
- type LineNumberStrip
- type LogCompressor
- type LogConfig
- type MarkdownStructured
- type TabularCompressor
Constants ¶
const MDBlockSentinel = "MDBLKX"
MDBlockSentinel is the line that stands in for one verbatim block in the masked document sent to the prose model. Requirements: single wordpiece-shaped token (like fidelity.py's FKEEPX), never plausibly present in real docs, and registered in the Python sidecar's FORCE_TOKENS (belt) while fidelity.protect also masks it as an ALL-CAPS entity (suspenders) and EXCLUDES it from identifier-density accounting (86 sentinels must not trigger identifier_dense).
Variables ¶
This section is empty.
Functions ¶
func ChainsWithModel ¶
func ChainsWithModel(modelURL string) map[compress.ContentType][]compress.Compressor
ChainsWithModel wires the per-type chains. modelURL=="" omits the prose and doc_read chains entirely (that content passes through untouched).
func DefaultChains ¶
func DefaultChains() map[compress.ContentType][]compress.Compressor
DefaultChains wires the per-type compressor chains. Types with no entry (code, diff, html, search, tabular, unknown) pass through (the pipeline reports them as skipped/no_compressor).
TypeTabular is deliberately reduced to the LOSSLESS ANSIStrip only - NewTabularCompressor (row-dropping) is NOT wired. Unlike JSON, a table's headers aren't repeated, so there is no lossless key-hoisting win to prefer; tabular's only real lever on raw text was lossy row-dropping, and silent row loss misleads an agent the same way JSON truncation did. On real developer-agent output the tabular route was also ~95% misrouted line-numbered file reads (now caught by detectLineNumbered) with near-zero genuine tables. Keeping ANSIStrip means a colored table is still de-ANSI'd losslessly and a plain table passes through untouched (ANSIStrip is a no-op → the expansion guardrail skips it) - never row-dropped. NewTabularCompressor is kept + tested for a future domain-gated, lossless-only reintroduction; it must never silently drop rows if re-wired. DefaultChains wires chains from the environment: deterministic compressors always; the ML prose path only when WHITTLE_MODEL_URL points at a running model sidecar (see model/). Whittle works out of the box without it.
Types ¶
type ANSIStrip ¶
type ANSIStrip struct{}
ANSIStrip removes ANSI escape codes and collapses runs of blank lines. It is deterministic and lossless for visible text. Used as the first link in the log, tabular, terminal, json and prose chains (any tool output may be colored). The escape stripping itself is delegated to compress.StripANSI so detection and compression share one definition of "an escape".
For TERMINAL-classified content ONLY it additionally collapses carriage-return overwrite chains (progress bars, spinners: `frame1\rframe2\r...`) to what the terminal actually displayed - CR returns the cursor to column 0 and subsequent output overwrites in place, so earlier frames were never persistently visible. Lossless by terminal-emulation semantics (measured 99% on a progress stream, docs/compressor-opportunities.md #4). Gated to TypeTerminal so a lone \r inside prose/JSON/log data is never touched.
func NewANSIStrip ¶
func NewANSIStrip() ANSIStrip
type JSONCrusher ¶
type JSONCrusher struct {
// contains filtered or unexported fields
}
JSONCrusher reduces JSON losslessly - it never drops rows:
- Minify (lossless, any JSON): json.Compact strips insignificant whitespace.
- Columnar reshape (lossless, uniform top-level array of objects): factor the repeated keys out into {"__schema__":[keys],"__rows__":[[vals],...]}. Every row survives; values are kept as raw JSON so types/nesting are exact and reconstruction is obj_i = {schema[j]: rows[i][j]}.
If nothing beats the input, passthrough (the pipeline's expansion guardrail then skips it).
The LOSSY representative-sampling path (first 30% ∪ last 15% ∪ one per distinct value, capped at maxItems) is currently DISABLED - see the TODO in Compress. Its helpers (selectItems/ceilPct/canonical, savingsRatio, losslessMinSavings, the maxItems field) are kept intact so re-enabling is just uncommenting.
func NewJSONCrusher ¶
func NewJSONCrusher() JSONCrusher
func (JSONCrusher) Handles ¶
func (JSONCrusher) Handles(ct compress.ContentType) bool
func (JSONCrusher) Name ¶
func (JSONCrusher) Name() string
type LLMLinguaAdapter ¶
type LLMLinguaAdapter struct {
// contains filtered or unexported fields
}
LLMLinguaAdapter delegates the prose path to the existing Python LLMLingua service. It sends content_class:"prose" so the Python gate compresses without re-gating. Any error / non-200 / action!=compressed is returned as an error so the pipeline fails open (passthrough). The client is reused across calls.
func NewLLMLinguaAdapterWithURL ¶
func NewLLMLinguaAdapterWithURL(url string) *LLMLinguaAdapter
func (*LLMLinguaAdapter) Handles ¶
func (*LLMLinguaAdapter) Handles(ct compress.ContentType) bool
func (*LLMLinguaAdapter) Name ¶
func (*LLMLinguaAdapter) Name() string
type LineNumberStrip ¶
type LineNumberStrip struct{}
LineNumberStrip removes `N\t` line-number prefixes from a doc_read (a line-numbered file read the router positively classified as a markdown/prose document) before the prose model runs. The prefixes are Read-tool render furniture, pure token cost, and would skew llmlingua's token scoring.
Deterministic; the numbering is mechanically re-derivable (lines are numbered contiguously), but the downstream prose model is lossy anyway, so this chain is lossy end-to-end by design - it only runs on content the router proved is a doc.
TODO(textcompress): CCR-style recovery - when the recovery store lands, stash the ORIGINAL doc_read content under a content hash and append a retrieval marker, so an agent that needs the exact file (or its line numbers, e.g. to quote a line reference) can recover it. Deferred with the other CCR work; see docs/smartcrusher-gap-analysis.md and docs/compressor-opportunities.md (#1).
func NewLineNumberStrip ¶
func NewLineNumberStrip() LineNumberStrip
func (LineNumberStrip) Handles ¶
func (LineNumberStrip) Handles(ct compress.ContentType) bool
func (LineNumberStrip) Name ¶
func (LineNumberStrip) Name() string
type LogCompressor ¶
type LogCompressor struct {
// contains filtered or unexported fields
}
LogCompressor keeps the signal in build/test/runtime logs: errors, fails, deduped warnings, stack traces and summary lines (plus a line of neighbor context), dropping the INFO/DEBUG noise. Ported from Headroom log_compressor.
func NewLogCompressor ¶
func NewLogCompressor(cfg LogConfig) LogCompressor
func (LogCompressor) Handles ¶
func (LogCompressor) Handles(ct compress.ContentType) bool
func (LogCompressor) Name ¶
func (LogCompressor) Name() string
type LogConfig ¶
type LogConfig struct {
MaxErrors int
MaxWarnings int
MaxStackTraces int
StackTraceMaxLines int
MaxTotalLines int
}
LogConfig bounds the selection. Zero values are NOT defaults - use DefaultLogConfig.
func DefaultLogConfig ¶
func DefaultLogConfig() LogConfig
type MarkdownStructured ¶
type MarkdownStructured struct {
// contains filtered or unexported fields
}
MarkdownStructured compresses a markdown doc_read STRUCTURE-AWARE: verbatim blocks (fenced/indented code, headings, tables, lists, quotes, HTML, link defs, config stragglers - everything compress.SegmentMarkdown marks verbatim) are swapped for MDBlockSentinel lines; ONLY the prose+sentinel document goes to the LLMLingua sidecar (one call - latency stays flat); the sentinels are then restored to the original blocks BYTE-EXACT, positionally.
Fail-open everywhere: sentinel collision in the input, a lost/duplicated sentinel in the model output, or a sidecar skip/error all return the original content as a clean skip. Restore is count-checked (the fidelity.py restore design): extractive models delete in order, so the i-th surviving sentinel is block i; any count mismatch aborts rather than risking a mis-splice.
This is what lets the router admit real technical docs (README/API/DOCS.md full of code fences) to doc_read: their code is protected HERE, structurally, not by refusing the whole document.
TODO(textcompress): CCR-style recovery for the original doc (line numbers + uncompressed prose) once the recovery store lands - see linenumstrip.go TODO.
func NewMarkdownStructuredWith ¶
func NewMarkdownStructuredWith(llm *LLMLinguaAdapter) *MarkdownStructured
NewMarkdownStructuredWith allows injecting the model adapter (tests).
func (*MarkdownStructured) Handles ¶
func (*MarkdownStructured) Handles(ct compress.ContentType) bool
func (*MarkdownStructured) Name ¶
func (*MarkdownStructured) Name() string
type TabularCompressor ¶
type TabularCompressor struct {
// contains filtered or unexported fields
}
TabularCompressor compresses column-aligned and delimited tables (kubectl/docker/ ls -l, psql, csv, markdown). Mirrors JSONCrusher's two layers: a LOSSLESS baseline that strips cosmetic alignment padding (the tabular analog of JSON minify), then optional row sampling for long tables that keeps the header plus a representative head/tail subset and marks the omitted middle. Headroom has a fixed-width parser but never wires a compressor to it; this is that missing piece.
func NewTabularCompressor ¶
func NewTabularCompressor() TabularCompressor
func (TabularCompressor) Handles ¶
func (TabularCompressor) Handles(ct compress.ContentType) bool
func (TabularCompressor) Name ¶
func (TabularCompressor) Name() string
Source Files