compact

Documentation

Overview

Package compact provides lossless source-code compaction for reducing token usage when feeding code to AI models.

It strips comments and removes blank lines from Go, Python, TypeScript, JavaScript, and Rust source files, producing valid, semantically-identical output. For Go files the standard AST parser is used so the result is always syntactically correct. For other languages a string-aware state machine correctly skips quoted literals before stripping comments.

Index

• func CompactSource(src []byte, lang Language) ([]byte, error)
• type Language
  • func DetectLanguage(filename string) Language
• type Stats
  • func CompactDir(dir, outDir string) (Stats, error)
  • func (s Stats) ByteReduction() float64
  • func (s Stats) CompactedTokens() int
  • func (s Stats) OriginalTokens() int
  • func (s Stats) String() string
  • func (s Stats) TokenReduction() float64

Functions

func CompactSource

func CompactSource(src []byte, lang Language) ([]byte, error)

CompactSource removes comments and blank lines from src.

For Go it uses the standard AST parser so the result is guaranteed to be syntactically valid Go. Compiler directives (//go:build, //go:generate, //go:embed, // +build) are preserved because they affect compilation.

For all other languages a string-aware state machine strips // and /* */ comments (or # comments for Python) while correctly skipping content inside quoted string literals, then removes blank lines.

Returns an error only for Go source that cannot be parsed.

Types

type Language

type Language string

Language is a recognised source language.

const (
	Go         Language = "go"
	Python     Language = "python"
	TypeScript Language = "typescript"
	JavaScript Language = "javascript"
	Rust       Language = "rust"
	Unknown    Language = ""
)

Supported language values for Language.

func DetectLanguage

func DetectLanguage(filename string) Language

DetectLanguage returns the Language for the given filename, or Unknown.

type Stats

type Stats struct {
	Files          int
	OriginalBytes  int
	CompactedBytes int
}

Stats records the aggregate result of a compaction run.

func CompactDir

func CompactDir(dir, outDir string) (Stats, error)

CompactDir walks dir and compacts every recognised source file, writing results to outDir. If outDir is empty files are written back in place. Unrecognised file types are skipped (not copied).

func (Stats) ByteReduction

func (s Stats) ByteReduction() float64

ByteReduction returns the percentage of bytes removed (0–100).

func (Stats) CompactedTokens

func (s Stats) CompactedTokens() int

CompactedTokens returns the estimated token count after compaction.

func (Stats) OriginalTokens

func (s Stats) OriginalTokens() int

OriginalTokens returns the estimated token count of the original source.

func (Stats) String

func (s Stats) String() string

String returns a human-readable summary.

func (Stats) TokenReduction

func (s Stats) TokenReduction() float64

TokenReduction returns the estimated percentage of LLM tokens removed. Token count is approximated as bytes / 4.