Documentation
¶
Index ¶
- Variables
- type TextEngine
- func (e *TextEngine) ChunkerFor(fileSize int64) chunker.Chunker
- func (e *TextEngine) DetectByExtension(path string) bool
- func (e *TextEngine) DetectByHeuristic(path string, header []byte) bool
- func (e *TextEngine) DetectByMagic(header []byte) bool
- func (e *TextEngine) Diff(ctx context.Context, oldPath string, oldReader io.Reader, newPath string, ...) (string, error)
- func (e *TextEngine) Metadata() *core.FileMetadata
- func (e *TextEngine) Name() string
- func (e *TextEngine) Preview(header []byte, size int64, reader io.Reader, maxLines int) (string, error)
Constants ¶
This section is empty.
Variables ¶
var ErrLineTooLong = errors.New("line too long for text diff")
ErrLineTooLong is returned by readLines when a single line exceeds the scanner's maximum buffer size (scannerMaxBufSize). Callers should treat this as "files differ" rather than a hard error, since the file is likely binary or minified and cannot be meaningfully line-diffed.
Functions ¶
This section is empty.
Types ¶
type TextEngine ¶
type TextEngine struct{}
TextEngine handles text files.
func (*TextEngine) ChunkerFor ¶
func (e *TextEngine) ChunkerFor(fileSize int64) chunker.Chunker
ChunkerFor selects the chunking strategy for a text file. Files smaller than textWholeFileThreshold return nil, signalling the caller to store the whole file as a single chunk. Larger files use FastCDC with the text-tiered defaults (4/8/16 KB — tuned for source code and structured text).
func (*TextEngine) DetectByExtension ¶
func (e *TextEngine) DetectByExtension(path string) bool
DetectByExtension checks if the path's extension or basename is a known text type.
func (*TextEngine) DetectByHeuristic ¶
func (e *TextEngine) DetectByHeuristic(path string, header []byte) bool
DetectByHeuristic is the last-resort content sniffing used for extensionless or unknown-extension files. A header is treated as text only when ALL of the following hold:
- it is non-empty and at least 4 bytes long,
- it does not start with a known image/video magic signature,
- it contains no NUL bytes (0x00),
- its control-byte ratio is at most 10%.
Control bytes are 0x01-0x1F (excluding \t, \n, \r) and 0x7F (DEL). High bytes (0x80-0xFF) are NOT counted as control bytes because valid UTF-8 text legitimately contains them. The 10% threshold catches raw binary data that happens to omit 0x00 (e.g. byte sequences 1..255, which are ~11% control bytes) while allowing text with occasional control characters.
Headers shorter than 4 bytes are too short for reliable heuristic detection (a single 0xFF byte would pass the control-ratio check because 0x80-0xFF are not counted as control bytes). Let the fallback binary engine handle them instead.
func (*TextEngine) DetectByMagic ¶
func (e *TextEngine) DetectByMagic(header []byte) bool
DetectByMagic checks file header signatures. Plain text has no unified magic byte signature, but files beginning with a Byte Order Mark (BOM) — UTF-8 (EF BB BF), UTF-16 big-endian (FE FF), or UTF-16 little-endian (FF FE) — are unambiguously text.
func (*TextEngine) Diff ¶
func (e *TextEngine) Diff(ctx context.Context, oldPath string, oldReader io.Reader, newPath string, newReader io.Reader) (string, error)
Diff produces a unified diff between two text files using Myers diff. Content is read streaming from oldReader/newReader via a line scanner; the Myers algorithm still requires both line arrays in memory, but the file bytes are never buffered whole. The context is threaded into the Myers/Hirschberg loops so a cancelled context aborts the diff promptly.
Inputs exceeding maxDiffLines fall back to a simple "files differ" message to avoid the O(N*M) Myers DP, which would be prohibitively slow for very large files. Identical content is short-circuited before the threshold check so that large but unchanged files produce no diff.
func (*TextEngine) Metadata ¶
func (e *TextEngine) Metadata() *core.FileMetadata
Metadata returns the file metadata for text files.
func (*TextEngine) Preview ¶
func (e *TextEngine) Preview(header []byte, size int64, reader io.Reader, maxLines int) (string, error)
Preview returns the first maxLines of text content, read streaming from reader. Only the requested number of lines are consumed, so previewing a huge text file does not load it whole.
Source Files