compression

package
v0.13.0-go Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// MinDynamicFrequency is the minimum number of times a phrase must
	// appear to be considered for the dynamic codebook.
	MinDynamicFrequency = 3
	// MinDynamicPhraseLen is the minimum character length for a phrase
	// to be eligible for dynamic compression.
	MinDynamicPhraseLen = 8
	// MaxDynamicCodes is the maximum number of dynamic codes to generate.
	MaxDynamicCodes = 20
	// DynamicCodePrefix is the prefix for dynamically generated codes.
	DynamicCodePrefix = "$X"
)
View Source 
const (
	// MaxToolResultLength is the maximum character length for a tool result
	// before it gets truncated.
	MaxToolResultLength = 2000
	// MaxToolResultLines is the maximum number of lines kept from a tool result.
	MaxToolResultLines = 50
	// TruncationMarker is appended when content is truncated.
	TruncationMarker = "\n...[truncated]"
)
View Source 
const (
	// MinPathSegments is the minimum number of path segments required
	// for a string to be considered a filesystem path.
	MinPathSegments = 3
	// MinPrefixOccurrences is the minimum number of times a path prefix
	// must appear to warrant replacement with a short variable.
	MinPrefixOccurrences = 3
	// MaxPathVariables is the maximum number of path variables ($P1-$P5).
	MaxPathVariables = 5
)
View Source 
const (
	// ShouldCompressThreshold is the minimum total character count across
	// all messages before compression is triggered.
	ShouldCompressThreshold = 5000
)

Variables

View Source 
var STATIC_CODEBOOK = map[string]string{

	"$OC01": "dosrouter",
	"$OC02": "tool_call",
	"$OC03": "function",
	"$OC04": "arguments",
	"$OC05": "assistant",
	"$OC06": "observation",
	"$OC07": "tool_result",
	"$OC08": "system_prompt",
	"$OC09": "user_message",
	"$OC10": "thought_process",

	"$SK01": "web_search",
	"$SK02": "code_execution",
	"$SK03": "file_operation",
	"$SK04": "knowledge_base",

	"$T01": "parameters",
	"$T02": "description",
	"$T03": "required",
	"$T04": "properties",
	"$T05": "string",
	"$T06": "boolean",
	"$T07": "integer",

	"$D01": "undefined",
	"$D02": "null",

	"$I01": "instructions",
	"$I02": "conversation",
	"$I03": "context_window",
	"$I04": "token_count",
	"$I05": "max_tokens",

	"$S01": "successfully",
	"$S02": "completed",
	"$S03": "error_message",

	"$J01": "json_schema",
	"$J02": "object",
	"$J03": "array",

	"$H01": "request",
	"$H02": "response",

	"$R01": "result",
	"$R02": "content",
	"$R03": "message",
	"$R04": "metadata",

	"$E01": "exception",
	"$E02": "traceback",
	"$E03": "error",
	"$E04": "warning",

	"$M01": "information",
	"$M02": "configuration",
	"$M03": "environment",
	"$M04": "application",
	"$M05": "implementation",
}

STATIC_CODEBOOK maps short codes to their expanded phrases. These codes are injected into compressed content and decoded by the receiving end.

Functions

func CompactJSON 

func CompactJSON(text string) string

CompactJSON takes a JSON string and re-encodes it without any extra whitespace. If the input is not valid JSON it is returned unchanged.

func CompressJSON 

func CompressJSON(messages []NormalizedMessage) int

CompressJSON compacts JSON found in tool_call arguments and tool message content. It modifies messages in-place and returns total characters saved.

func CompressObservations 

func CompressObservations(messages []NormalizedMessage) int

CompressObservations aggressively compresses tool result messages by extracting key information and truncating large results. It modifies messages in-place and returns total characters saved.

func CompressWhitespace 

func CompressWhitespace(messages []NormalizedMessage) int

CompressWhitespace applies whitespace normalization to all messages. Returns the total characters saved.

func DecompressContent 

func DecompressContent(text string, extraCodes map[string]string) string

DecompressContent replaces all codebook tokens in text with their expanded phrases. It handles both static and provided extra codes.

func GenerateCodebookHeader 

func GenerateCodebookHeader(usedCodes []string, extraCodes map[string]string) string

GenerateCodebookHeader produces a human-readable header listing all codes that were actually used during compression plus any extra codes.

func GetInverseCodebook 

func GetInverseCodebook() map[string]string

GetInverseCodebook returns a map from phrase -> code.

func NormalizeWhitespace 

func NormalizeWhitespace(text string) string

NormalizeWhitespace applies whitespace normalization to a string:

  • CRLF -> LF
  • Limit consecutive newlines to 2
  • Trim trailing spaces per line
  • Collapse multiple spaces (mid-line) to single
  • Reduce deep indentation (>8 spaces) to 2-space levels
  • Tabs -> 2 spaces

func ShouldCompress 

func ShouldCompress(messages []NormalizedMessage) bool

ShouldCompress returns true if the total text content across all messages exceeds ShouldCompressThreshold characters.

Types

type CompressionConfig 

type CompressionConfig struct {
	Dedup           bool `json:"dedup"`
	Whitespace      bool `json:"whitespace"`
	Dictionary      bool `json:"dictionary"`
	Paths           bool `json:"paths"`
	JSONCompact     bool `json:"jsonCompact"`
	Observation     bool `json:"observation"`
	DynamicCodebook bool `json:"dynamicCodebook"`
}

CompressionConfig controls which compression layers are applied.

func DefaultCompressionConfig 

func DefaultCompressionConfig() CompressionConfig

DefaultCompressionConfig returns the default configuration with only dedup, whitespace, and jsonCompact enabled.

type CompressionResult 

type CompressionResult struct {
	Messages []NormalizedMessage `json:"messages"`
	Stats    CompressionStats    `json:"stats"`
	Header   string              `json:"header,omitempty"`
}

CompressionResult is the output of CompressContext.

func CompressContext 

func CompressContext(messages []NormalizedMessage, config CompressionConfig) CompressionResult

CompressContext applies all enabled compression layers to a slice of messages according to the provided config and returns a CompressionResult.

Layer execution order:

  1. Deduplication (remove duplicate assistant messages)
  2. Whitespace normalization
  3. JSON compaction (tool args + tool results)
  4. Observation compression (truncate large tool results)
  5. Dictionary compression (static codebook substitution)
  6. Path compression (common path prefix replacement)
  7. Dynamic codebook (frequency-based phrase compression)

After all layers run, a codebook header is prepended to the first user message so the model can decode any short codes.

type CompressionStats 

type CompressionStats struct {
	OriginalChars   int               `json:"originalChars"`
	CompressedChars int               `json:"compressedChars"`
	Ratio           float64           `json:"ratio"`
	LayerSavings    map[string]int    `json:"layerSavings"`
	MessagesRemoved int               `json:"messagesRemoved"`
	DictCodesUsed   []string          `json:"dictCodesUsed,omitempty"`
	PathMappings    map[string]string `json:"pathMappings,omitempty"`
}

CompressionStats tracks metrics about the compression process.

type ContentPart 

type ContentPart struct {
	Type     ContentPartType        `json:"type"`
	Text     string                 `json:"text,omitempty"`
	ImageURL map[string]interface{} `json:"image_url,omitempty"`
}

ContentPart represents a single part of a multi-part message content.

type ContentPartType 

type ContentPartType string

ContentPartType identifies the kind of content in a multi-part message. 

const (
	ContentPartTypeText     ContentPartType = "text"
	ContentPartTypeImageURL ContentPartType = "image_url"
)

type DictionaryResult 

type DictionaryResult struct {
	UsedCodes  []string
	CharsSaved int
}

DictionaryResult holds the outcome of dictionary compression.

func CompressDictionary 

func CompressDictionary(messages []NormalizedMessage) DictionaryResult

CompressDictionary replaces known phrases from the static codebook with their short codes. Phrases are replaced longest-first so that longer matches take priority over substrings.

It modifies messages in-place and returns stats about which codes were used and how many characters were saved.

type DynamicCodebookResult 

type DynamicCodebookResult struct {
	Codes      map[string]string // code -> phrase
	CharsSaved int
}

DynamicCodebookResult holds the outcome of dynamic codebook compression.

func CompressDynamicCodebook 

func CompressDynamicCodebook(messages []NormalizedMessage) DynamicCodebookResult

CompressDynamicCodebook analyzes content across all messages to find frequently repeated phrases, assigns short dynamic codes ($X01-$X20), and replaces them in the content.

It modifies messages in-place and returns the generated codebook and characters saved.

type NormalizedMessage 

type NormalizedMessage struct {
	Role       string        `json:"role"`
	Content    string        `json:"content,omitempty"`
	Parts      []ContentPart `json:"parts,omitempty"`
	ToolCallID string        `json:"tool_call_id,omitempty"`
	ToolCalls  []ToolCall    `json:"tool_calls,omitempty"`
	Name       string        `json:"name,omitempty"`
}

NormalizedMessage is the unified message format used throughout the compression pipeline. Content can be either a plain string or a slice of ContentPart for multi-modal messages.

func DeduplicateMessages 

func DeduplicateMessages(messages []NormalizedMessage) ([]NormalizedMessage, int)

DeduplicateMessages removes duplicate assistant messages while preserving system, user, and tool messages unconditionally. Assistant messages that have tool_calls referenced by subsequent tool messages are also kept.

Returns the deduplicated slice and the number of messages removed.

func PrependCodebookHeader 

func PrependCodebookHeader(messages []NormalizedMessage, header string) []NormalizedMessage

PrependCodebookHeader inserts a codebook header before the content of the first user message. This allows the model to decode compressed codes.

func (*NormalizedMessage) GetTextContent 

func (m *NormalizedMessage) GetTextContent() string

GetTextContent returns the text content of the message. For multi-part messages it concatenates all text parts.

func (*NormalizedMessage) SetTextContent 

func (m *NormalizedMessage) SetTextContent(text string)

SetTextContent sets the text content. If the message uses Parts, it updates the first text part; otherwise it sets Content directly.

type PathsResult 

type PathsResult struct {
	Mappings   map[string]string
	CharsSaved int
}

PathsResult holds the outcome of path compression.

func CompressPaths 

func CompressPaths(messages []NormalizedMessage) PathsResult

CompressPaths extracts filesystem paths from all messages, identifies common prefixes appearing 3+ times, and replaces them with $P1-$P5 variables. It modifies messages in-place.

type ToolCall 

type ToolCall struct {
	ID       string           `json:"id"`
	Type     string           `json:"type"`
	Function ToolCallFunction `json:"function"`
}

ToolCall represents a tool invocation within an assistant message.

type ToolCallFunction 

type ToolCallFunction struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCallFunction holds the function name and arguments for a tool call.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.