Documentation
¶
Overview ¶
Package trim is query-aware context compression: given a context and the current ask it returns a verbatim subset of the context's lines, in their original order, with one short marker line where lines were removed. It never rewrites a line.
Each line is scored by a small embedded gradient-boosted tree model over lexical, structural and positional features (LineFeatures); an operating point turns the scores into a selection. Pure Go, deterministic, no network and no files at runtime. It fails closed: an unknown operating point, an unusable model, a blank query, a small or oversized input, a line that reads as a marker or a cut too small to matter returns the context unchanged.
Index ¶
Constants ¶
const DefaultPoint = "balanced"
DefaultPoint is used when Options.Point is empty.
Variables ¶
var FeatureNames = []string{
"bm25", "bm25_norm", "bm25_rank", "bm25_win_sum", "cover_win_sum", "bm25_tail_norm",
"cover", "cover_idf", "cover_n", "cover_maxidf", "cover_rank", "cover_tail",
"compound_hits", "compound_maxlen", "bigram_hits", "trigram_hits",
"idf_mean", "idf_max", "uniq_frac",
"k_error", "k_test", "k_path", "k_pathline", "k_heading", "k_diff", "k_section",
"k_signature", "k_jsonkey", "k_separator", "k_frame", "k_kv", "k_bullet",
"dup", "tmpl", "tmpl_first",
"pos_rel", "pos_head", "pos_tail", "n_lines", "ctx_chars",
"len_chars", "len_words", "alnum_frac", "upper_frac", "digit_frac",
"indent", "indent_delta", "block_start", "header_dist", "header_bm25", "header_cover",
"nb_bm25_max", "nb_bm25_mean", "nb_cover_max", "nb_compound_max", "prev_bm25", "next_bm25",
"prev_blank", "next_blank",
"q_words", "q_terms", "q_found", "ctx_bm25_max",
"para_cover_max", "sec_cover", "prf_cover", "prf_compound",
}
FeatureNames lists the per-line features in the order LineFeatures writes them. The embedded model was trained on exactly this order; training and inference both call LineFeatures, so there is one code path.
var Points = map[string]Policy{
"conservative": {Floor: 0.0035, TopK: 3, Context: 1, Headers: 1, Gap: 2, MinChars: 2000, MinCut: 0.1},
"balanced": {Floor: 0.008, TopK: 3, Context: 1, Headers: 1, Gap: 2, MinChars: 2000, MinCut: 0.1},
"aggressive": {Floor: 0.035, TopK: 3, Context: 1, Headers: 1, Gap: 2, MinChars: 2000, MinCut: 0.1},
}
Points are the named operating points, tuned on the benchmark dev splits.
Functions ¶
func LineFeatures ¶
LineFeatures splits context at "\n" and returns its lines with one feature row per line, nFeatures wide. Only the query's last maxQueryBytes count. The result depends only on the inputs: map lookups never decide an order and every float sum runs in line order.
Types ¶
type Options ¶
type Options struct {
// Point names an entry of Points; empty means DefaultPoint. An unknown
// name returns the context unchanged.
Point string
// Policy, when set, replaces the named point (for tuning and frontiers).
Policy *Policy
}
Options selects how much to keep.
type Policy ¶
type Policy struct {
// Floor keeps every line scoring at least this share of the context's
// best line. Relative, because calibrated probabilities shrink as a
// context grows: one absolute threshold cuts long contexts far too hard.
Floor float64
// TopK always keeps the K best-scoring lines, so the output is never only
// markers.
TopK int
// Context keeps this many lines on each side of a line kept by Floor or
// TopK.
Context int
// Headers keeps up to this many enclosing block headers (the nearest
// earlier lines with a smaller indent) of every such line and, when above
// zero, the line opening its section (a Markdown heading, "[start of
// path]", "==> path <==", "diff --git", "File: path", or any short line
// in square brackets, such as a dated "[Session 8: ...]").
Headers int
// Gap fills a run of at most this many removed lines between kept lines;
// a marker would cost about as much as the lines.
Gap int
// MinChars returns contexts shorter than this unchanged.
MinChars int
// MinCut returns the context unchanged when the selection would remove
// less than this share of its bytes.
MinCut float64
}
Policy turns line scores into a selection.
type Result ¶
type Result struct {
// Output is the compressed text, or the context itself when Unchanged.
Output string
// KeptLines and RemovedLines count the context's lines ("\n" separated;
// the empty piece after a final "\n" is not a line).
KeptLines, RemovedLines int
// Point is the operating point used ("custom" for Options.Policy).
Point string
Policy Policy
// Unchanged is true when the context was returned as is; Reason says why.
Unchanged bool
Reason string
}
Result is the outcome of one Compress call.