compaction

Documentation

Index

• Constants
• Variables
• func EstimateMessageTokens(msg *chat.Message) int64
• func FirstIndexInBudget(messages []chat.Message, contextLimit int64) int
• func ShouldCompact(inputTokens, outputTokens, addedTokens, contextLimit int64, threshold float64) bool
• func SplitIndexForKeep(messages []chat.Message, maxTokens int64) int
• type Estimator
  • func NewEstimator(messages iter.Seq[*chat.Message]) Estimator
  • func NewSliceEstimator(messages []chat.Message) Estimator
  • func (e Estimator) EstimateMessageTokens(msg *chat.Message) int64
  • func (e Estimator) Scale() float64

Constants

const DefaultThreshold = 0.9

DefaultThreshold is the fraction of the context window at which compaction is triggered when no custom threshold is configured. When the estimated token usage exceeds this fraction of the context limit, compaction is recommended. Overridable per agent (and per model) via the `compaction_threshold` config key.

Variables

var (
	//go:embed prompts/compaction-system.txt
	SystemPrompt string

	//go:embed prompts/compaction-user.txt
	UserPrompt string
)

Functions

func EstimateMessageTokens

func EstimateMessageTokens(msg *chat.Message) int64

EstimateMessageTokens returns a token estimate for a single chat message: the provider-reported count when the message carries usage data (assistant turns), otherwise a chars-per-token heuristic that is intentionally conservative (overestimates) so proactive compaction fires before we hit the limit.

This is the uncalibrated entry point; use NewEstimator when a conversation with provider-reported usage is available to reconcile the heuristic against.

func FirstIndexInBudget

func FirstIndexInBudget(messages []chat.Message, contextLimit int64) int

FirstIndexInBudget returns the smallest index N such that messages[N:] fits within contextLimit, snapping to a user/assistant turn boundary. Used to truncate the conversation handed to the summarization model so the request itself doesn't blow the context window. Like SplitIndexForKeep it sizes messages with an Estimator calibrated on the slice.

When the entire slice fits within contextLimit, the function returns the index of the earliest user/assistant message in the suffix — older tool-only messages (which can't legally start a conversation) are dropped. In the unusual case of a tool-only conversation with no user/asst turns, it returns len(messages); callers should treat that as "nothing to send" and skip the truncation.

func ShouldCompact

func ShouldCompact(inputTokens, outputTokens, addedTokens, contextLimit int64, threshold float64) bool

ShouldCompact reports whether a session's context usage has crossed the compaction threshold. It returns true when the total token count (input + output + addedTokens) exceeds threshold*contextLimit.

threshold is the fraction of the context window that triggers compaction; values outside (0, 1] (including 0 for "not configured") fall back to DefaultThreshold, so callers can pass an unset value verbatim.

func SplitIndexForKeep

func SplitIndexForKeep(messages []chat.Message, maxTokens int64) int

SplitIndexForKeep walks messages from the end and returns the earliest index whose suffix fits in maxTokens, snapping to user/assistant boundaries. All messages from the returned index onward are intended to be preserved verbatim across a compaction; messages before it are the candidates to summarize. Returns len(messages) when everything fits in the keep budget — i.e. compact everything.

Token sizes come from an Estimator calibrated on the same slice, so assistant turns weigh their provider-reported counts and the rest is heuristic corrected by observed usage.

The boundary snap matters for providers (notably Anthropic) that reject conversations starting on a tool-result message: by stopping the kept window on a user/assistant turn, we guarantee the kept suffix begins on a clean conversational turn.

Types

type Estimator

type Estimator struct {
	// contains filtered or unexported fields
}

Estimator estimates message token counts, reconciling the chars-per-token heuristic with provider-reported usage observed in a conversation. The zero value is a neutral estimator (no calibration).

func NewEstimator

func NewEstimator(messages iter.Seq[*chat.Message]) Estimator

NewEstimator derives an Estimator calibrated against the provider-reported usage found in a conversation.

Every assistant message that carries usage is an anchor whose prompt tokens are the provider's exact count of everything before it. For two consecutive anchors i < j:

prompt(j) − (prompt(i) + output(i))

is the provider-tokenized size of the messages between them (tool results and user turns — precisely the content the heuristic has to guess at). System-prompt and tool-definition overhead cancels out of the delta. The ratio of summed deltas to the summed heuristic estimates of those in-between messages becomes a multiplicative correction applied to heuristic estimates.

Guard rails, biased toward compacting slightly early rather than overflowing:

• windows with a non-positive delta are discarded — a compaction between the two anchors rebuilt the prompt from a summary, so the delta no longer measures the in-between messages;
• windows whose anchors were produced by different models are discarded — the delta would mix two tokenizers;
• the anchor's reasoning tokens are excluded from its total, since providers strip reasoning from subsequent prompts (when they do resend it, the delta only grows, erring conservative);
• in-between messages with their own reported counts contribute to neither sum: their exact size is subtracted from the delta so the ratio stays a pure heuristic-vs-provider comparison;
• transient context injected per turn (hook output) inflates deltas slightly, erring on the conservative side;
• the ratio is trusted only when backed by [calibrationMinTokens] of heuristic mass and is clamped to [calibrationMin, calibrationMax].

func NewSliceEstimator

func NewSliceEstimator(messages []chat.Message) Estimator

NewSliceEstimator is a convenience wrapper around NewEstimator for a slice of messages.

func (Estimator) EstimateMessageTokens

func (e Estimator) EstimateMessageTokens(msg *chat.Message) int64

EstimateMessageTokens returns the token estimate for one message: the provider-reported count when available (exact, never scaled), otherwise the calibrated heuristic.

func (Estimator) Scale

func (e Estimator) Scale() float64

Scale returns the multiplier applied to heuristic estimates: 1 for a neutral estimator, >1 when provider reports showed the heuristic underestimating this conversation's content, <1 when it overestimated.