normalize

Documentation

Index

• Constants
• func HasValidExt(path string) bool
• func HashLine(raw []byte, fn func(hash uint64, chunk []byte) bool)
• func HashReader(r io.Reader, emit func(h uint64, rawLine []byte), buf []byte) int
• func IsValidUtf8(path string) bool
• func NewScanBuf() []byte
• func PathHash(p string) uint64

Constants

const MaxTokenSize = 1024 * 1024 * 10 // 10 MB

Functions

func HasValidExt

func HasValidExt(path string) bool

HasValidExt reports whether path has a recognized code file extension.

func HashLine

func HashLine(raw []byte, fn func(hash uint64, chunk []byte) bool)

HashLine normalizes a line, then hashes it (chunking if minified). Calls fn for each hash produced with the chunk that produced it. For non-chunked lines, chunk == the full normalized line. fn returns true to continue, false to stop. Empty/comment lines produce no calls to fn.

func HashReader

func HashReader(r io.Reader, emit func(h uint64, rawLine []byte), buf []byte) int

HashReader scans lines from r, normalizes and hashes each line. For each hash produced, emit is called with the hash and the raw scanner line. Returns the total number of hashes processed. If buf is non-nil, it is used as the scanner buffer (see NewScanBuf).

func IsValidUtf8

func IsValidUtf8(path string) bool

IsValidUtf8 checks if the first 8KB of a file is valid UTF-8.

func NewScanBuf

func NewScanBuf() []byte

NewScanBuf allocates a reusable scanner buffer. Pass it to HashReader to avoid a 10 MB allocation per call. Safe to reuse across sequential calls.

func PathHash

func PathHash(p string) uint64

PathHash returns the hash for a path entry (prefixed with "path:").