encoding

Documentation

Overview

BOM-driven UTF transcoding at the pipeline's byte gateway. Files written in UTF-16 or UTF-32 are converted to UTF-8 in one place so every downstream pass can stay UTF-8-only. Byte order marks are removed on the way in and never re-emitted - the output side always emits UTF-8 without a BOM.

Index

• Variables
• func NormalizeToUTF8(src []byte) ([]byte, error)

Variables

var (
	BOMUTF8    = []byte{0xEF, 0xBB, 0xBF}
	BOMUTF16BE = []byte{0xFE, 0xFF}
	BOMUTF16LE = []byte{0xFF, 0xFE}
	BOMUTF32BE = []byte{0x00, 0x00, 0xFE, 0xFF}
	BOMUTF32LE = []byte{0xFF, 0xFE, 0x00, 0x00}
)

Well-known byte-order marks. Order-of-check matters at call sites: UTF-32 LE starts with the UTF-16 LE prefix, so the longer marker must be tested first.

Functions

func NormalizeToUTF8

func NormalizeToUTF8(src []byte) ([]byte, error)

Detects an encoding-signaling BOM and, when the input isn't UTF-8, transcodes the payload to UTF-8 on the fly so the rest of the pipeline can stay UTF-8-only. Output is always UTF-8 - a file that arrived as UTF-16 leaves as UTF-8, which is the modern default and what most YAML consumers expect anyway.