textfile

Documentation

Overview

Package textfile owns charset and line-ending detection/conversion at the boundary between a file's on-disk bytes and nib's in-memory, always-UTF-8-with-LF representation — the same split internal/textwidth makes for display-width math. Nothing outside a Buffer's Load/Save should need to know a file's charset or EOL style; everywhere else, text is plain UTF-8 split on "\n".

Index

• func Encode(text string, charset Charset) ([]byte, error)
• func JoinLines(lines []string, eol EOL) string
• type Charset
  • func Decode(data []byte) (text string, charset Charset, err error)
• type EOL
  • func SplitLines(text string) (lines []string, eol EOL)

Functions

func Encode

func Encode(text string, charset Charset) ([]byte, error)

Encode is Decode's inverse: re-adds a BOM and/or re-encodes to UTF-16 as charset requires. UTF8 (the zero value) is a plain byte-slice cast, identical to today's Save.

func JoinLines

func JoinLines(lines []string, eol EOL) string

JoinLines is SplitLines' inverse for a given EOL style.

Types

type Charset

type Charset string

Charset identifies how a file's bytes are encoded. The zero value, UTF8, is deliberately today's default (no BOM) — every Buffer{} literal built without setting Charset (there are many, across tests) keeps behaving exactly as before.

const (
	UTF8    Charset = ""
	UTF8BOM Charset = "utf-8-bom"
	UTF16LE Charset = "utf-16le"
	UTF16BE Charset = "utf-16be"
)

func Decode

func Decode(data []byte) (text string, charset Charset, err error)

Decode converts data to a UTF-8 string, detecting its charset from a byte-order mark. With no recognized BOM, data is assumed to already be UTF-8 — today's behavior, unchanged: an invalid byte sequence is not an error, it just decodes to the Unicode replacement character wherever it occurs (Go's string() conversion already does this for free).

type EOL

type EOL string

EOL identifies a file's line-ending style. The zero value, LF, is deliberately today's default (a bare "\n"), for the same reason as Charset's zero value above.

const (
	LF   EOL = ""
	CRLF EOL = "crlf"
	CR   EOL = "cr"
)

func SplitLines

func SplitLines(text string) (lines []string, eol EOL)

SplitLines detects text's line-ending style (the first "\r\n", bare "\r", or "\n" wins; no terminator at all defaults to LF) and splits it into lines with every line-ending sequence stripped — including a lone trailing one, so an all-empty text is a single empty line. Mixed line endings within one text are all treated as line breaks; only the detected (first) style is reported back, to use on a subsequent JoinLines.

For an LF-only text (the common case) this produces byte-for-byte the same lines as the old TrimSuffix+Split it replaces.