fileops

Documentation

Overview

Package fileops provides file system operations for token counting, including directory traversal with .gitignore support and binary detection.

Index

• func AggregateFileContents(ctx context.Context, files []string) ([]byte, error)
• func IsBinaryContent(path string, content []byte) bool
• func IsBinaryFile(path string, collectors ...BinaryStatsCollector) (bool, error)
• type BinaryStatsCollector
• type WalkResult
  • func WalkDirectory(ctx context.Context, rootPath string, collectors ...WalkStatsCollector) (*WalkResult, error)
• type WalkStatsCollector

Functions

func AggregateFileContents

func AggregateFileContents(ctx context.Context, files []string) ([]byte, error)

AggregateFileContents reads all files and returns combined content. Pre-allocates the result buffer based on file sizes to minimize allocations.

func IsBinaryContent

func IsBinaryContent(path string, content []byte) bool

IsBinaryContent applies the same extension and null-byte prefix rules as IsBinaryFile to bytes that have already been read by a caller.

func IsBinaryFile

func IsBinaryFile(path string, collectors ...BinaryStatsCollector) (bool, error)

IsBinaryFile checks if a file is likely binary.

Types

type BinaryStatsCollector

type BinaryStatsCollector interface {
	RecordBinarySniffOpen()
	RecordBinarySniffBytes(int64)
	RecordValidationReadDuration(time.Duration)
}

BinaryStatsCollector receives optional instrumentation from binary detection. It is deliberately small so the normal path can pass nil without creating a dependency on the tokenizer package.

type WalkResult

type WalkResult struct {
	Files         []string
	TotalFiles    int
	SkippedBinary int
	SkippedIgnore int
}

WalkResult contains information about walked files.

func WalkDirectory

func WalkDirectory(ctx context.Context, rootPath string, collectors ...WalkStatsCollector) (*WalkResult, error)

WalkDirectory recursively walks a directory, respecting .gitignore files and filtering out binary files.

type WalkStatsCollector

type WalkStatsCollector interface {
	BinaryStatsCollector
	RecordEntryVisited()
	RecordEligibleFile()
	RecordWalkDuration(time.Duration)
}

WalkStatsCollector receives optional instrumentation from directory walking.