encsniff

README

encsniff-go

A small Go library for sniffing common non-UTF-8 text encodings at the head of a file or byte slice. It detects only patterns with byte-perfect signatures — no heuristics. It returns an action (use as is, strip BOM, or warn) and a copy-pasteable iconv hint when conversion is needed.

Companion to encsniff (Rust).

Install

go get github.com/excelano/encsniff-go

Usage

import "github.com/excelano/encsniff-go"

s, err := encsniff.SniffFile("Roster_Report.csv")
if err != nil { /* ... */ }

switch s.Action {
case encsniff.UseAsIs:
    // proceed
case encsniff.StripBom:
    // skip s.BomLen bytes silently
case encsniff.Warn:
    fmt.Fprintf(os.Stderr, "warning: file appears to be %s encoded.\n", s.Encoding)
    fmt.Fprintf(os.Stderr, "hint: %s\n", s.Hint)
}

SniffBytes(b []byte) Sniff is the in-memory version.

What it detects

Pattern	Action	Why
EF BB BF at offset 0	StripBom	UTF-8 BOM from "Save as CSV UTF-8". Skip the 3 bytes; the file is otherwise clean.
FF FE at offset 0	Warn	UTF-16 little-endian. Hint suggests iconv -f UTF-16LE -t UTF-8.
FE FF at offset 0	Warn	UTF-16 big-endian. Hint suggests iconv -f UTF-16BE -t UTF-8.
+ACI- in first 4KB	Warn	UTF-7 escape for " (common in Scoutbook and some Microsoft exports). Hint suggests iconv -f UTF-7 -t UTF-8.
Window is not valid UTF-8	WarnUnknown	The bytes are decidably not UTF-8, but which encoding they are is not knowable from a signature. encoding is empty. Hint suggests trying iconv -f WINDOWS-1252 or -f LATIN1.
Anything else	UseAsIs	Assume UTF-8/ASCII; no guessing.

WarnUnknown is not an exception to the no-guessing rule. Naming a single-byte encoding would be a guess; saying the bytes are not UTF-8 is not, because UTF-8 is a decidable grammar. The verdict is "definitely not UTF-8, and I cannot tell you which encoding it is" — and the hint that comes with it is worded as something to try rather than as a claim about the file.

The scan window is 4 KB, so a character straddling its end arrives truncated. Truncation is never reported as invalid: only a byte that could not appear where it did counts. The cost of that rule is one quiet case — a file ending on a lone lead byte reads as clean, because it is indistinguishable from a character cut in half. Real legacy exports put high bytes in front of ASCII repeatedly, so they are flagged on the first such pair.

A clean sniff still proves nothing about the rest of the file. This has always been a head-of-file check, so callers still need a decent error on the read path and should not treat UseAsIs as a guarantee.

What it does not do

No heuristic encoding detection. CP1252 vs Latin-1, language-based detection, byte-frequency analysis are all out of scope. If you need that, reach for uchardet.

License

MIT. Author: David M. Anderson. Built with AI assistance (Claude, Anthropic).

Documentation

Overview

Package encsniff detects common non-UTF-8 text encodings from the head of a file or byte slice, using only byte-perfect signatures — no heuristics.

It returns one of four actions: UseAsIs (clean UTF-8/ASCII), StripBom (UTF-8 BOM that should be silently skipped), Warn (a named non-UTF-8 encoding the caller should surface along with an iconv hint), or WarnUnknown (bytes that are provably not UTF-8, encoding unnameable).

Index

• func IconvCommand(enc Encoding, path string) string
• func IconvGuessCommand(path string) string
• type Action
• type Encoding
• type Sniff
  • func SniffBytes(b []byte) Sniff
  • func SniffFile(path string) (Sniff, error)
  • func (s Sniff) IsWarning() bool

Functions

func IconvCommand

func IconvCommand(enc Encoding, path string) string

IconvCommand composes an iconv command that converts path from enc to UTF-8, writing to a sibling file with .utf8 inserted before the extension. It returns the empty string for encodings that need no conversion.

func IconvGuessCommand

func IconvGuessCommand(path string) string

IconvGuessCommand composes a suggested conversion for a file that is provably not UTF-8 but whose encoding could not be named.

Deliberately worded as something to try rather than as a claim. Every other hint this package produces follows a byte-perfect signature and can assert what the file is; this one follows only the absence of valid UTF-8, and the two candidates it names are the common cases, not a detection.

Types

type Action

type Action int

Action is what the caller should do with the file.

const (
	// UseAsIs means the input looks like UTF-8 or ASCII; proceed unchanged.
	UseAsIs Action = iota
	// StripBom means the input is UTF-8 with a leading BOM; skip BomLen bytes.
	StripBom
	// Warn means the input is a non-UTF-8 encoding the user should know about.
	Warn
	// WarnUnknown means the input is provably not UTF-8, but which encoding it
	// is could not be determined. Encoding is empty — nothing was proven about
	// the identity, only about what it is not.
	//
	// Naming a single-byte encoding would be a guess, and this package does not
	// guess. Saying the bytes are not UTF-8 is not a guess: UTF-8 is a decidable
	// grammar, and this decides it exactly.
	WarnUnknown
)

type Encoding

type Encoding string

Encoding names the detected encoding. The zero value is empty.

const (
	EncUTF8    Encoding = "UTF-8"
	EncUTF8BOM Encoding = "UTF-8 with BOM"
	EncUTF16LE Encoding = "UTF-16 little-endian"
	EncUTF16BE Encoding = "UTF-16 big-endian"
	EncUTF7    Encoding = "UTF-7"
)

type Sniff

type Sniff struct {
	// Action tells the caller what to do.
	Action Action
	// Encoding is the detected encoding (empty if UseAsIs and no BOM).
	Encoding Encoding
	// BomLen is the number of leading bytes to skip when Action == StripBom.
	BomLen int
	// Hint is a copy-pasteable iconv command. Set by SniffFile on either warning
	// action; empty when sniffing bytes with no associated path.
	Hint string
}

Sniff is the result of a detection pass.

func SniffBytes

func SniffBytes(b []byte) Sniff

SniffBytes inspects the head of b and returns the detected action. The Hint field is left empty; callers with a path should use SniffFile, or compose a hint with IconvCommand.

func SniffFile

func SniffFile(path string) (Sniff, error)

SniffFile opens path, sniffs the head, and returns the result with Hint set to a copy-pasteable iconv command when Action == Warn.

func (Sniff) IsWarning

func (s Sniff) IsWarning() bool

IsWarning reports whether this result is something the user should be told about.

Prefer this to comparing against Warn: every consumer in the fleet wrote `s.Action != encsniff.Warn` to mean "nothing to report", and in Go a new constant does not break that — it is silently ignored, which is worse than a compile error. A predicate absorbs the next verdict too.