charset

Documentation

Overview

Package charset converts the character encodings that actually appear in syndicated feeds to UTF-8. Go's encoding/xml only reads UTF-8, but a large share of real-world RSS still declares iso-8859-1 or windows-1252, ships a UTF-16 byte-order mark, or lies about being UTF-8 while containing raw windows-1252 bytes. This package repairs all of those cases with a small decoding table instead of an external encodings dependency.

Index

• func DecodeWindows1252(data []byte) []byte
• func Reader(label string, input io.Reader) (io.Reader, error)
• func RepairUTF8(data []byte) ([]byte, bool)
• func SanitizeXML(data []byte) ([]byte, bool)
• func ToUTF8(data []byte) ([]byte, string)

Functions

func DecodeWindows1252

func DecodeWindows1252(data []byte) []byte

DecodeWindows1252 converts windows-1252 (and therefore also iso-8859-1) bytes to UTF-8.

func Reader

func Reader(label string, input io.Reader) (io.Reader, error)

Reader is an xml.Decoder CharsetReader covering the encodings that appear in real feeds. iso-8859-1 is deliberately decoded as windows-1252: the latter is a superset in practice, and feeds declaring latin-1 routinely contain smart quotes from the 0x80-0x9F range (the same superset rule browsers apply).

func RepairUTF8

func RepairUTF8(data []byte) ([]byte, bool)

RepairUTF8 replaces invalid UTF-8 sequences. When a document declares UTF-8 but contains windows-1252 bytes (the classic copy-paste smart-quote bug), reinterpreting the invalid bytes as windows-1252 recovers the intended characters; sequences are never silently dropped.

func SanitizeXML

func SanitizeXML(data []byte) ([]byte, bool)

SanitizeXML removes control characters that are illegal in XML 1.0 (everything below 0x20 except tab, newline, and carriage return). Real feeds contain stray 0x00 and 0x1A bytes pasted from editors, and encoding/xml aborts on them. Returns the cleaned bytes and whether any byte was removed.

func ToUTF8

func ToUTF8(data []byte) ([]byte, string)

ToUTF8 handles byte-order marks up front: it strips a UTF-8 BOM and transcodes UTF-16 (either endianness) to UTF-8. This must happen before XML parsing because encoding/xml cannot even read the <?xml?> declaration of a UTF-16 document. The second return value names the applied conversion, or "" when the data was passed through untouched.