encsniff

README

encsniff-go

A small Go library for sniffing common non-UTF-8 text encodings at the head of a file or byte slice. It detects only patterns with byte-perfect signatures — no heuristics. It returns an action (use as is, strip BOM, or warn) and a copy-pasteable iconv hint when conversion is needed.

Companion to encsniff (Rust).

Install

go get github.com/excelano/encsniff-go

Usage

import "github.com/excelano/encsniff-go"

s, err := encsniff.SniffFile("Roster_Report.csv")
if err != nil { /* ... */ }

switch s.Action {
case encsniff.UseAsIs:
    // proceed
case encsniff.StripBom:
    // skip s.BomLen bytes silently
case encsniff.Warn:
    fmt.Fprintf(os.Stderr, "warning: file appears to be %s encoded.\n", s.Encoding)
    fmt.Fprintf(os.Stderr, "hint: %s\n", s.Hint)
}

SniffBytes(b []byte) Sniff is the in-memory version.

What it detects

Pattern	Action	Why
EF BB BF at offset 0	StripBom	UTF-8 BOM from "Save as CSV UTF-8". Skip the 3 bytes; the file is otherwise clean.
FF FE at offset 0	Warn	UTF-16 little-endian. Hint suggests iconv -f UTF-16LE -t UTF-8.
FE FF at offset 0	Warn	UTF-16 big-endian. Hint suggests iconv -f UTF-16BE -t UTF-8.
+ACI- in first 4KB	Warn	UTF-7 escape for " (common in Scoutbook and some Microsoft exports). Hint suggests iconv -f UTF-7 -t UTF-8.
Anything else	UseAsIs	Assume UTF-8/ASCII; no guessing.

What it does not do

No heuristic encoding detection. CP1252 vs Latin-1, language-based detection, byte-frequency analysis are all out of scope. If you need that, reach for uchardet.

License

MIT. Author: David M. Anderson. Built with AI assistance (Claude, Anthropic).

Documentation

Overview

Package encsniff detects common non-UTF-8 text encodings from the head of a file or byte slice, using only byte-perfect signatures — no heuristics.

It returns one of three actions: UseAsIs (clean UTF-8/ASCII), StripBom (UTF-8 BOM that should be silently skipped), or Warn (a non-UTF-8 encoding the caller should surface to the user along with an iconv hint).

Index

• func IconvCommand(enc Encoding, path string) string
• type Action
• type Encoding
• type Sniff
  • func SniffBytes(b []byte) Sniff
  • func SniffFile(path string) (Sniff, error)

Functions

func IconvCommand

func IconvCommand(enc Encoding, path string) string

IconvCommand composes an iconv command that converts path from enc to UTF-8, writing to a sibling file with .utf8 inserted before the extension. It returns the empty string for encodings that need no conversion.

Types

type Action

type Action int

Action is what the caller should do with the file.

const (
	// UseAsIs means the input looks like UTF-8 or ASCII; proceed unchanged.
	UseAsIs Action = iota
	// StripBom means the input is UTF-8 with a leading BOM; skip BomLen bytes.
	StripBom
	// Warn means the input is a non-UTF-8 encoding the user should know about.
	Warn
)

type Encoding

type Encoding string

Encoding names the detected encoding. The zero value is empty.

const (
	EncUTF8    Encoding = "UTF-8"
	EncUTF8BOM Encoding = "UTF-8 with BOM"
	EncUTF16LE Encoding = "UTF-16 little-endian"
	EncUTF16BE Encoding = "UTF-16 big-endian"
	EncUTF7    Encoding = "UTF-7"
)

type Sniff

type Sniff struct {
	// Action tells the caller what to do.
	Action Action
	// Encoding is the detected encoding (empty if UseAsIs and no BOM).
	Encoding Encoding
	// BomLen is the number of leading bytes to skip when Action == StripBom.
	BomLen int
	// Hint is a copy-pasteable iconv command. Set by SniffFile when Action ==
	// Warn; empty when sniffing bytes with no associated path.
	Hint string
}

Sniff is the result of a detection pass.

func SniffBytes

func SniffBytes(b []byte) Sniff

SniffBytes inspects the head of b and returns the detected action. The Hint field is left empty; callers with a path should use SniffFile, or compose a hint with IconvCommand.

func SniffFile

func SniffFile(path string) (Sniff, error)

SniffFile opens path, sniffs the head, and returns the result with Hint set to a copy-pasteable iconv command when Action == Warn.