utfbom

README

utfbom

utfbom detects and removes a leading Unicode byte order mark (BOM) from an io.Reader. It can also report the encoding indicated by that BOM.

The remaining bytes pass through unchanged. The package does not decode text, convert UTF-16 or UTF-32 to UTF-8, validate Unicode, or guess an encoding when there is no BOM.

Installation

go get github.com/dimchansky/utfbom

Requires Go 1.19 or later. No dependencies outside the standard library.

Example

package main

import (
	"fmt"
	"io"
	"strings"

	"github.com/dimchansky/utfbom"
)

func main() {
	r, enc := utfbom.Skip(strings.NewReader("\xEF\xBB\xBFhello"))

	data, err := io.ReadAll(r)
	if err != nil {
		panic(err)
	}

	fmt.Printf("%s %q\n", enc, data) // UTF8 "hello"
}

If you don't need the encoding, use SkipOnly:

data, err := io.ReadAll(utfbom.SkipOnly(file))

Supported BOMs

Encoding	BOM bytes	Constant
UTF-8	EF BB BF	UTF8
UTF-16 big-endian	FE FF	UTF16BigEndian
UTF-16 little-endian	FF FE	UTF16LittleEndian
UTF-32 big-endian	00 00 FE FF	UTF32BigEndian
UTF-32 little-endian	FF FE 00 00	UTF32LittleEndian

FF FE 00 00 is reported as UTF-32 little-endian, although the same bytes could be a UTF-16 little-endian BOM followed by a NUL character.

Behavior

• At most one BOM is removed: the one at the current position of the reader you pass in. BOM-like bytes later in the stream are data and stay as they are.
• Unknown means no supported BOM was detected. Plain UTF-8 text without a BOM is also Unknown.
• After the BOM is removed, UTF-16 and UTF-32 data is still UTF-16 and UTF-32.
• Skip and SkipOnly read up to 4 bytes from the source before returning, so they can block.
• An error that occurs while reading those bytes is returned once by Read, after the bytes that remain after BOM removal. Later calls to Read go to the underlying reader.
• If probing stops with a non-EOF error (including io.ErrNoProgress after 100 consecutive reads returning (0, nil)) before 4 bytes are read, a UTF-16 BOM (FE FF / FF FE) is not recognized: the encoding is Unknown and the bytes are passed through. Complete UTF-8 and UTF-32 BOMs are still recognized.
• Calling Skip on a *utfbom.Reader returns that same reader and the encoding it detected. This does not apply to other readers that wrap a *utfbom.Reader.
• The returned reader does not close the source; close it yourself.

Development

make check   # go vet, golangci-lint, tests with -race
make fmt     # format the code

make lint and make fmt need golangci-lint v2.15.0 or later.

License

Apache License 2.0

Documentation

Overview

Package utfbom implements the detection of the BOM (Unicode Byte Order Mark) and removing as necessary. It wraps an io.Reader object, creating another object (Reader) that also implements the io.Reader interface but provides automatic BOM checking and removing as necessary.

Index

• func Skip(rd io.Reader) (*Reader, Encoding)
• type Encoding
  • func (e Encoding) String() string
• type Reader
  • func SkipOnly(rd io.Reader) *Reader
  • func (r *Reader) Read(p []byte) (n int, err error)

Examples

• Skip
• Skip (Utf16)
• SkipOnly

Functions

func Skip

func Skip(rd io.Reader) (*Reader, Encoding)

Skip returns a Reader that reads from rd without its leading BOM (Unicode Byte Order Mark), and the encoding indicated by that BOM, or Unknown if no supported BOM was detected.

Skip reads up to 4 bytes from rd before it returns, so it may block. An error that occurs while reading them is not returned by Skip: the returned Reader's Read returns it once, after the bytes that remain after BOM removal.

If rd is already a *Reader, Skip returns rd itself and the encoding it detected, without reading from it. Use SkipOnly if the encoding is not needed.

Example

package main

import (
	"fmt"
	"io"
	"strings"

	"github.com/dimchansky/utfbom"
)

func main() {
	r, enc := utfbom.Skip(strings.NewReader("\xEF\xBB\xBFhello"))

	data, err := io.ReadAll(r)
	if err != nil {
		panic(err)
	}

	fmt.Printf("%s %q\n", enc, data)
}

Output:
UTF8 "hello"

Example (Utf16)

The BOM is removed, but the remaining bytes stay UTF-16: nothing is transcoded.

package main

import (
	"fmt"
	"io"
	"strings"

	"github.com/dimchansky/utfbom"
)

func main() {
	r, enc := utfbom.Skip(strings.NewReader("\xFF\xFEh\x00i\x00"))

	data, err := io.ReadAll(r)
	if err != nil {
		panic(err)
	}

	fmt.Printf("%s % X\n", enc, data)
}

Output:
UTF16LittleEndian 68 00 69 00

Types

type Encoding

type Encoding int

Encoding is a UTF encoding identified by a BOM.

const (
	// Unknown encoding, returned when no BOM was detected
	Unknown Encoding = iota

	// UTF8, BOM bytes: EF BB BF
	UTF8

	// UTF-16, big-endian, BOM bytes: FE FF
	UTF16BigEndian

	// UTF-16, little-endian, BOM bytes: FF FE
	UTF16LittleEndian

	// UTF-32, big-endian, BOM bytes: 00 00 FE FF
	UTF32BigEndian

	// UTF-32, little-endian, BOM bytes: FF FE 00 00
	UTF32LittleEndian
)

Constants to identify detected UTF encodings.

func (Encoding) String

func (e Encoding) String() string

String returns a user-friendly string representation of the encoding. Satisfies fmt.Stringer interface.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader is an io.Reader that returns the bytes of the underlying reader without the leading BOM (Unicode Byte Order Mark). Create it with Skip or SkipOnly. Reader does not close the underlying reader.

func SkipOnly

func SkipOnly(rd io.Reader) *Reader

SkipOnly is like Skip but does not return the encoding. It also reads from rd immediately and may block, and an error from that read is returned later by Read.

Example

package main

import (
	"fmt"
	"io"
	"strings"

	"github.com/dimchansky/utfbom"
)

func main() {
	data, err := io.ReadAll(utfbom.SkipOnly(strings.NewReader("\xEF\xBB\xBFhello")))
	if err != nil {
		panic(err)
	}

	fmt.Printf("%q\n", data)
}

Output:
"hello"

func (*Reader) Read

func (r *Reader) Read(p []byte) (n int, err error)

Read implements io.Reader. The BOM is detected once, by Skip or SkipOnly: Read returns the bytes read past the BOM during detection, then any error hit during detection (once), then reads from the underlying reader directly.