text

package module

v1.6.0 Latest Latest Go to latest Published: Mar 26, 2023 License: MIT Imports: 12 Imported by: 3

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/mithrandie/go-text

Links

Open Source Insights

README ¶

go-text

The package text and the sub-packages are Go libraries to operate text.

Requirements

Go 1.18 or later.

Supported Character Encodings

UTF-8
UTF-16
Shift-JIS

Sub Packages

color : This package provides support for coloring text with ANSI escape sequences.

csv : This package provides support for reading and writing CSV format.

fixedlen : This package provides support for reading and writing Fixed-Length format.

json : This package provides support for encoding and decoding JSON format.

jsonl : This package provides support for reading JSON Lines format.

ltsv : This package provides support for reading and writing LTSV format.

table : This package provides support for writing text tables.

Documentation ¶

Overview ¶

Package text is a Go library to operate text.

Calculation of string width is conformed to https://www.unicode.org/reports/tr11/

Index ¶

Constants
Variables
func ByteSize(s string, encoding Encoding) int
func Decode(src []byte, enc Encoding) ([]byte, error)
func Encode(src []byte, enc Encoding) ([]byte, error)
func GetTransformDecoder(r io.Reader, enc Encoding) (io.Reader, error)
func GetTransformEncoder(r io.Reader, enc Encoding) (io.Reader, error)
func GetTransformWriter(w io.Writer, enc Encoding) (io.Writer, error)
func IsHighSurrogate(r rune) bool
func IsLowSurrogate(r rune) bool
func IsRightToLeftLetters(s string) bool
func NewUTF8MEncoder() *encoding.Encoder
func RuneByteSize(r rune, encoding Encoding) int
func RuneWidth(r rune, eastAsianEncoding bool, countDiacriticalSign bool, ...) int
func Width(s string, eastAsianEncoding bool, countDiacriticalSign bool, ...) int
type Encoding
- func DetectEncoding(r io.ReadSeeker) (detected Encoding, err error)
- func DetectInSpecifiedEncoding(r io.ReadSeeker, enc Encoding) (detected Encoding, err error)
- func InferEncoding(b []byte, eof bool) (Encoding, error)
- func ParseEncoding(s string) (Encoding, error)
- func (e Encoding) String() string
type FieldAlignment
type LineBreak
- func ParseLineBreak(s string) (LineBreak, error)
- func (lb LineBreak) String() string
- func (lb LineBreak) Value() string
type RawText
type UTF8MEncoder
- func (u *UTF8MEncoder) Reset()
- func (u *UTF8MEncoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error)

Constants ¶

View Source

const (
	UTF8BOM    = "\xef\xbb\xbf"
	UTF16BEBOM = "\xfe\xff"
	UTF16LEBOM = "\xff\xfe"
)

View Source

const (
	SurrogatePairHighBegin = "\xd8\x00"
	SurrogatePairHighEnd   = "\xdb\xff"
	SurrogatePairLowBegin  = "\xdc\x00"
	SurrogatePairLowEnd    = "\xdf\xff"
)

Variables ¶

View Source

var AmbiguousTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x00a1, Hi: 0x00a1, Stride: 1},
		{Lo: 0x00a4, Hi: 0x00a4, Stride: 1},
		{Lo: 0x00a7, Hi: 0x00a7, Stride: 1},
		{Lo: 0x00a8, Hi: 0x00a8, Stride: 1},
		{Lo: 0x00aa, Hi: 0x00aa, Stride: 1},
		{Lo: 0x00ad, Hi: 0x00ad, Stride: 1},
		{Lo: 0x00ae, Hi: 0x00ae, Stride: 1},
		{Lo: 0x00b0, Hi: 0x00b0, Stride: 1},
		{Lo: 0x00b1, Hi: 0x00b1, Stride: 1},
		{Lo: 0x00b2, Hi: 0x00b3, Stride: 1},
		{Lo: 0x00b4, Hi: 0x00b4, Stride: 1},
		{Lo: 0x00b6, Hi: 0x00b7, Stride: 1},
		{Lo: 0x00b8, Hi: 0x00b8, Stride: 1},
		{Lo: 0x00b9, Hi: 0x00b9, Stride: 1},
		{Lo: 0x00ba, Hi: 0x00ba, Stride: 1},
		{Lo: 0x00bc, Hi: 0x00be, Stride: 1},
		{Lo: 0x00bf, Hi: 0x00bf, Stride: 1},
		{Lo: 0x00c6, Hi: 0x00c6, Stride: 1},
		{Lo: 0x00d0, Hi: 0x00d0, Stride: 1},
		{Lo: 0x00d7, Hi: 0x00d7, Stride: 1},
		{Lo: 0x00d8, Hi: 0x00d8, Stride: 1},
		{Lo: 0x00de, Hi: 0x00e1, Stride: 1},
		{Lo: 0x00e6, Hi: 0x00e6, Stride: 1},
		{Lo: 0x00e8, Hi: 0x00ea, Stride: 1},
		{Lo: 0x00ec, Hi: 0x00ed, Stride: 1},
		{Lo: 0x00f0, Hi: 0x00f0, Stride: 1},
		{Lo: 0x00f2, Hi: 0x00f3, Stride: 1},
		{Lo: 0x00f7, Hi: 0x00f7, Stride: 1},
		{Lo: 0x00f8, Hi: 0x00fa, Stride: 1},
		{Lo: 0x00fc, Hi: 0x00fc, Stride: 1},
		{Lo: 0x00fe, Hi: 0x00fe, Stride: 1},
		{Lo: 0x0101, Hi: 0x0101, Stride: 1},
		{Lo: 0x0111, Hi: 0x0111, Stride: 1},
		{Lo: 0x0113, Hi: 0x0113, Stride: 1},
		{Lo: 0x011b, Hi: 0x011b, Stride: 1},
		{Lo: 0x0126, Hi: 0x0127, Stride: 1},
		{Lo: 0x012b, Hi: 0x012b, Stride: 1},
		{Lo: 0x0131, Hi: 0x0133, Stride: 1},
		{Lo: 0x0138, Hi: 0x0138, Stride: 1},
		{Lo: 0x013f, Hi: 0x0142, Stride: 1},
		{Lo: 0x0144, Hi: 0x0144, Stride: 1},
		{Lo: 0x0148, Hi: 0x014b, Stride: 1},
		{Lo: 0x014d, Hi: 0x014d, Stride: 1},
		{Lo: 0x0152, Hi: 0x0153, Stride: 1},
		{Lo: 0x0166, Hi: 0x0167, Stride: 1},
		{Lo: 0x016b, Hi: 0x016b, Stride: 1},
		{Lo: 0x01ce, Hi: 0x01ce, Stride: 1},
		{Lo: 0x01d0, Hi: 0x01d0, Stride: 1},
		{Lo: 0x01d2, Hi: 0x01d2, Stride: 1},
		{Lo: 0x01d4, Hi: 0x01d4, Stride: 1},
		{Lo: 0x01d6, Hi: 0x01d6, Stride: 1},
		{Lo: 0x01d8, Hi: 0x01d8, Stride: 1},
		{Lo: 0x01da, Hi: 0x01da, Stride: 1},
		{Lo: 0x01dc, Hi: 0x01dc, Stride: 1},
		{Lo: 0x0251, Hi: 0x0251, Stride: 1},
		{Lo: 0x0261, Hi: 0x0261, Stride: 1},
		{Lo: 0x02c4, Hi: 0x02c4, Stride: 1},
		{Lo: 0x02c7, Hi: 0x02c7, Stride: 1},
		{Lo: 0x02c9, Hi: 0x02cb, Stride: 1},
		{Lo: 0x02cd, Hi: 0x02cd, Stride: 1},
		{Lo: 0x02d0, Hi: 0x02d0, Stride: 1},
		{Lo: 0x02d8, Hi: 0x02db, Stride: 1},
		{Lo: 0x02dd, Hi: 0x02dd, Stride: 1},
		{Lo: 0x02df, Hi: 0x02df, Stride: 1},
		{Lo: 0x0300, Hi: 0x036f, Stride: 1},
		{Lo: 0x0391, Hi: 0x03a1, Stride: 1},
		{Lo: 0x03a3, Hi: 0x03a9, Stride: 1},
		{Lo: 0x03b1, Hi: 0x03c1, Stride: 1},
		{Lo: 0x03c3, Hi: 0x03c9, Stride: 1},
		{Lo: 0x0401, Hi: 0x0401, Stride: 1},
		{Lo: 0x0410, Hi: 0x044f, Stride: 1},
		{Lo: 0x0451, Hi: 0x0451, Stride: 1},
		{Lo: 0x2010, Hi: 0x2010, Stride: 1},
		{Lo: 0x2013, Hi: 0x2015, Stride: 1},
		{Lo: 0x2016, Hi: 0x2016, Stride: 1},
		{Lo: 0x2018, Hi: 0x2018, Stride: 1},
		{Lo: 0x2019, Hi: 0x2019, Stride: 1},
		{Lo: 0x201c, Hi: 0x201c, Stride: 1},
		{Lo: 0x201d, Hi: 0x201d, Stride: 1},
		{Lo: 0x2020, Hi: 0x2022, Stride: 1},
		{Lo: 0x2024, Hi: 0x2027, Stride: 1},
		{Lo: 0x2030, Hi: 0x2030, Stride: 1},
		{Lo: 0x2032, Hi: 0x2033, Stride: 1},
		{Lo: 0x2035, Hi: 0x2035, Stride: 1},
		{Lo: 0x203b, Hi: 0x203b, Stride: 1},
		{Lo: 0x203e, Hi: 0x203e, Stride: 1},
		{Lo: 0x2074, Hi: 0x2074, Stride: 1},
		{Lo: 0x207f, Hi: 0x207f, Stride: 1},
		{Lo: 0x2081, Hi: 0x2084, Stride: 1},
		{Lo: 0x20ac, Hi: 0x20ac, Stride: 1},
		{Lo: 0x2103, Hi: 0x2103, Stride: 1},
		{Lo: 0x2105, Hi: 0x2105, Stride: 1},
		{Lo: 0x2109, Hi: 0x2109, Stride: 1},
		{Lo: 0x2113, Hi: 0x2113, Stride: 1},
		{Lo: 0x2116, Hi: 0x2116, Stride: 1},
		{Lo: 0x2121, Hi: 0x2122, Stride: 1},
		{Lo: 0x2126, Hi: 0x2126, Stride: 1},
		{Lo: 0x212b, Hi: 0x212b, Stride: 1},
		{Lo: 0x2153, Hi: 0x2154, Stride: 1},
		{Lo: 0x215b, Hi: 0x215e, Stride: 1},
		{Lo: 0x2160, Hi: 0x216b, Stride: 1},
		{Lo: 0x2170, Hi: 0x2179, Stride: 1},
		{Lo: 0x2189, Hi: 0x2189, Stride: 1},
		{Lo: 0x2190, Hi: 0x2194, Stride: 1},
		{Lo: 0x2195, Hi: 0x2199, Stride: 1},
		{Lo: 0x21b8, Hi: 0x21b9, Stride: 1},
		{Lo: 0x21d2, Hi: 0x21d2, Stride: 1},
		{Lo: 0x21d4, Hi: 0x21d4, Stride: 1},
		{Lo: 0x21e7, Hi: 0x21e7, Stride: 1},
		{Lo: 0x2200, Hi: 0x2200, Stride: 1},
		{Lo: 0x2202, Hi: 0x2203, Stride: 1},
		{Lo: 0x2207, Hi: 0x2208, Stride: 1},
		{Lo: 0x220b, Hi: 0x220b, Stride: 1},
		{Lo: 0x220f, Hi: 0x220f, Stride: 1},
		{Lo: 0x2211, Hi: 0x2211, Stride: 1},
		{Lo: 0x2215, Hi: 0x2215, Stride: 1},
		{Lo: 0x221a, Hi: 0x221a, Stride: 1},
		{Lo: 0x221d, Hi: 0x2220, Stride: 1},
		{Lo: 0x2223, Hi: 0x2223, Stride: 1},
		{Lo: 0x2225, Hi: 0x2225, Stride: 1},
		{Lo: 0x2227, Hi: 0x222c, Stride: 1},
		{Lo: 0x222e, Hi: 0x222e, Stride: 1},
		{Lo: 0x2234, Hi: 0x2237, Stride: 1},
		{Lo: 0x223c, Hi: 0x223d, Stride: 1},
		{Lo: 0x2248, Hi: 0x2248, Stride: 1},
		{Lo: 0x224c, Hi: 0x224c, Stride: 1},
		{Lo: 0x2252, Hi: 0x2252, Stride: 1},
		{Lo: 0x2260, Hi: 0x2261, Stride: 1},
		{Lo: 0x2264, Hi: 0x2267, Stride: 1},
		{Lo: 0x226a, Hi: 0x226b, Stride: 1},
		{Lo: 0x226e, Hi: 0x226f, Stride: 1},
		{Lo: 0x2282, Hi: 0x2283, Stride: 1},
		{Lo: 0x2286, Hi: 0x2287, Stride: 1},
		{Lo: 0x2295, Hi: 0x2295, Stride: 1},
		{Lo: 0x2299, Hi: 0x2299, Stride: 1},
		{Lo: 0x22a5, Hi: 0x22a5, Stride: 1},
		{Lo: 0x22bf, Hi: 0x22bf, Stride: 1},
		{Lo: 0x2312, Hi: 0x2312, Stride: 1},
		{Lo: 0x2460, Hi: 0x249b, Stride: 1},
		{Lo: 0x249c, Hi: 0x24e9, Stride: 1},
		{Lo: 0x24eb, Hi: 0x24ff, Stride: 1},
		{Lo: 0x2500, Hi: 0x254b, Stride: 1},
		{Lo: 0x2550, Hi: 0x2573, Stride: 1},
		{Lo: 0x2580, Hi: 0x258f, Stride: 1},
		{Lo: 0x2592, Hi: 0x2595, Stride: 1},
		{Lo: 0x25a0, Hi: 0x25a1, Stride: 1},
		{Lo: 0x25a3, Hi: 0x25a9, Stride: 1},
		{Lo: 0x25b2, Hi: 0x25b3, Stride: 1},
		{Lo: 0x25b6, Hi: 0x25b6, Stride: 1},
		{Lo: 0x25b7, Hi: 0x25b7, Stride: 1},
		{Lo: 0x25bc, Hi: 0x25bd, Stride: 1},
		{Lo: 0x25c0, Hi: 0x25c0, Stride: 1},
		{Lo: 0x25c1, Hi: 0x25c1, Stride: 1},
		{Lo: 0x25c6, Hi: 0x25c8, Stride: 1},
		{Lo: 0x25cb, Hi: 0x25cb, Stride: 1},
		{Lo: 0x25ce, Hi: 0x25d1, Stride: 1},
		{Lo: 0x25e2, Hi: 0x25e5, Stride: 1},
		{Lo: 0x25ef, Hi: 0x25ef, Stride: 1},
		{Lo: 0x2605, Hi: 0x2606, Stride: 1},
		{Lo: 0x2609, Hi: 0x2609, Stride: 1},
		{Lo: 0x260e, Hi: 0x260f, Stride: 1},
		{Lo: 0x261c, Hi: 0x261c, Stride: 1},
		{Lo: 0x261e, Hi: 0x261e, Stride: 1},
		{Lo: 0x2640, Hi: 0x2640, Stride: 1},
		{Lo: 0x2642, Hi: 0x2642, Stride: 1},
		{Lo: 0x2660, Hi: 0x2661, Stride: 1},
		{Lo: 0x2663, Hi: 0x2665, Stride: 1},
		{Lo: 0x2667, Hi: 0x266a, Stride: 1},
		{Lo: 0x266c, Hi: 0x266d, Stride: 1},
		{Lo: 0x266f, Hi: 0x266f, Stride: 1},
		{Lo: 0x269e, Hi: 0x269f, Stride: 1},
		{Lo: 0x26bf, Hi: 0x26bf, Stride: 1},
		{Lo: 0x26c6, Hi: 0x26cd, Stride: 1},
		{Lo: 0x26cf, Hi: 0x26d3, Stride: 1},
		{Lo: 0x26d5, Hi: 0x26e1, Stride: 1},
		{Lo: 0x26e3, Hi: 0x26e3, Stride: 1},
		{Lo: 0x26e8, Hi: 0x26e9, Stride: 1},
		{Lo: 0x26eb, Hi: 0x26f1, Stride: 1},
		{Lo: 0x26f4, Hi: 0x26f4, Stride: 1},
		{Lo: 0x26f6, Hi: 0x26f9, Stride: 1},
		{Lo: 0x26fb, Hi: 0x26fc, Stride: 1},
		{Lo: 0x26fe, Hi: 0x26ff, Stride: 1},
		{Lo: 0x273d, Hi: 0x273d, Stride: 1},
		{Lo: 0x2776, Hi: 0x277f, Stride: 1},
		{Lo: 0x2b56, Hi: 0x2b59, Stride: 1},
		{Lo: 0x3248, Hi: 0x324f, Stride: 1},
		{Lo: 0xe000, Hi: 0xf8ff, Stride: 1},
		{Lo: 0xfe00, Hi: 0xfe0f, Stride: 1},
		{Lo: 0xfffd, Hi: 0xfffd, Stride: 1},
	},
	R32: []unicode.Range32{
		{Lo: 0x1f100, Hi: 0x1f10a, Stride: 1},
		{Lo: 0x1f110, Hi: 0x1f12d, Stride: 1},
		{Lo: 0x1f130, Hi: 0x1f169, Stride: 1},
		{Lo: 0x1f170, Hi: 0x1f18d, Stride: 1},
		{Lo: 0x1f18f, Hi: 0x1f190, Stride: 1},
		{Lo: 0x1f19b, Hi: 0x1f1ac, Stride: 1},
		{Lo: 0xe0100, Hi: 0xe01ef, Stride: 1},
		{Lo: 0xf0000, Hi: 0xffffd, Stride: 1},
		{Lo: 0x100000, Hi: 0x10fffd, Stride: 1},
	},
}

View Source

var DiacriticalSignTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x0300, Hi: 0x036f, Stride: 1},
		{Lo: 0x0591, Hi: 0x05af, Stride: 1},
		{Lo: 0x05b0, Hi: 0x05bd, Stride: 1},
		{Lo: 0x05bf, Hi: 0x05bf, Stride: 1},
		{Lo: 0x05c1, Hi: 0x05c2, Stride: 1},
		{Lo: 0x05c4, Hi: 0x05c5, Stride: 1},
		{Lo: 0x05c7, Hi: 0x05c7, Stride: 1},
		{Lo: 0x064b, Hi: 0x0652, Stride: 1},
		{Lo: 0x0653, Hi: 0x065f, Stride: 1},
		{Lo: 0x0670, Hi: 0x0670, Stride: 1},
		{Lo: 0x08a0, Hi: 0x08ff, Stride: 1},
		{Lo: 0x1ab0, Hi: 0x1aff, Stride: 1},
		{Lo: 0x1dc0, Hi: 0x1dff, Stride: 1},
		{Lo: 0x20d0, Hi: 0x20ff, Stride: 1},
		{Lo: 0xfbb2, Hi: 0xfbc1, Stride: 1},
		{Lo: 0xfe20, Hi: 0xfe2f, Stride: 1},
	},
}

View Source

var EncodingLiteral = map[Encoding]string{
	AUTO:     "AUTO",
	UTF8:     "UTF8",
	UTF8M:    "UTF8M",
	UTF16:    "UTF16",
	UTF16BEM: "UTF16BEM",
	UTF16LEM: "UTF16LEM",
	UTF16BE:  "UTF16BE",
	UTF16LE:  "UTF16LE",
	SJIS:     "SJIS",
}

View Source

var ErrInvalidEncoding = errors.New("invalid character encoding")

View Source

var ErrUnknownEncoding = errors.New("cannot detect character encoding")

View Source

var FormatCharTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x034f, Hi: 0x034f, Stride: 1},
		{Lo: 0x200c, Hi: 0x200f, Stride: 1},
		{Lo: 0x2028, Hi: 0x202e, Stride: 1},
		{Lo: 0x2061, Hi: 0x2069, Stride: 1},
	},
}

View Source

var FullWidthTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x1100, Hi: 0x115f, Stride: 1},
		{Lo: 0x231a, Hi: 0x231b, Stride: 1},
		{Lo: 0x2329, Hi: 0x2329, Stride: 1},
		{Lo: 0x232a, Hi: 0x232a, Stride: 1},
		{Lo: 0x23e9, Hi: 0x23ec, Stride: 1},
		{Lo: 0x23f0, Hi: 0x23f0, Stride: 1},
		{Lo: 0x23f3, Hi: 0x23f3, Stride: 1},
		{Lo: 0x25fd, Hi: 0x25fe, Stride: 1},
		{Lo: 0x2614, Hi: 0x2615, Stride: 1},
		{Lo: 0x2648, Hi: 0x2653, Stride: 1},
		{Lo: 0x267f, Hi: 0x267f, Stride: 1},
		{Lo: 0x2693, Hi: 0x2693, Stride: 1},
		{Lo: 0x26a1, Hi: 0x26a1, Stride: 1},
		{Lo: 0x26aa, Hi: 0x26ab, Stride: 1},
		{Lo: 0x26bd, Hi: 0x26be, Stride: 1},
		{Lo: 0x26c4, Hi: 0x26c5, Stride: 1},
		{Lo: 0x26ce, Hi: 0x26ce, Stride: 1},
		{Lo: 0x26d4, Hi: 0x26d4, Stride: 1},
		{Lo: 0x26ea, Hi: 0x26ea, Stride: 1},
		{Lo: 0x26f2, Hi: 0x26f3, Stride: 1},
		{Lo: 0x26f5, Hi: 0x26f5, Stride: 1},
		{Lo: 0x26fa, Hi: 0x26fa, Stride: 1},
		{Lo: 0x26fd, Hi: 0x26fd, Stride: 1},
		{Lo: 0x2705, Hi: 0x2705, Stride: 1},
		{Lo: 0x270a, Hi: 0x270b, Stride: 1},
		{Lo: 0x2728, Hi: 0x2728, Stride: 1},
		{Lo: 0x274c, Hi: 0x274c, Stride: 1},
		{Lo: 0x274e, Hi: 0x274e, Stride: 1},
		{Lo: 0x2753, Hi: 0x2755, Stride: 1},
		{Lo: 0x2757, Hi: 0x2757, Stride: 1},
		{Lo: 0x2795, Hi: 0x2797, Stride: 1},
		{Lo: 0x27b0, Hi: 0x27b0, Stride: 1},
		{Lo: 0x27bf, Hi: 0x27bf, Stride: 1},
		{Lo: 0x2b1b, Hi: 0x2b1c, Stride: 1},
		{Lo: 0x2b50, Hi: 0x2b50, Stride: 1},
		{Lo: 0x2b55, Hi: 0x2b55, Stride: 1},
		{Lo: 0x2e80, Hi: 0x2e99, Stride: 1},
		{Lo: 0x2e9b, Hi: 0x2ef3, Stride: 1},
		{Lo: 0x2f00, Hi: 0x2fd5, Stride: 1},
		{Lo: 0x2ff0, Hi: 0x2ffb, Stride: 1},
		{Lo: 0x3000, Hi: 0x3000, Stride: 1},
		{Lo: 0x3001, Hi: 0x3003, Stride: 1},
		{Lo: 0x3004, Hi: 0x3004, Stride: 1},
		{Lo: 0x3005, Hi: 0x3005, Stride: 1},
		{Lo: 0x3006, Hi: 0x3006, Stride: 1},
		{Lo: 0x3007, Hi: 0x3007, Stride: 1},
		{Lo: 0x3008, Hi: 0x3008, Stride: 1},
		{Lo: 0x3009, Hi: 0x3009, Stride: 1},
		{Lo: 0x300a, Hi: 0x300a, Stride: 1},
		{Lo: 0x300b, Hi: 0x300b, Stride: 1},
		{Lo: 0x300c, Hi: 0x300c, Stride: 1},
		{Lo: 0x300d, Hi: 0x300d, Stride: 1},
		{Lo: 0x300e, Hi: 0x300e, Stride: 1},
		{Lo: 0x300f, Hi: 0x300f, Stride: 1},
		{Lo: 0x3010, Hi: 0x3010, Stride: 1},
		{Lo: 0x3011, Hi: 0x3011, Stride: 1},
		{Lo: 0x3012, Hi: 0x3013, Stride: 1},
		{Lo: 0x3014, Hi: 0x3014, Stride: 1},
		{Lo: 0x3015, Hi: 0x3015, Stride: 1},
		{Lo: 0x3016, Hi: 0x3016, Stride: 1},
		{Lo: 0x3017, Hi: 0x3017, Stride: 1},
		{Lo: 0x3018, Hi: 0x3018, Stride: 1},
		{Lo: 0x3019, Hi: 0x3019, Stride: 1},
		{Lo: 0x301a, Hi: 0x301a, Stride: 1},
		{Lo: 0x301b, Hi: 0x301b, Stride: 1},
		{Lo: 0x301c, Hi: 0x301c, Stride: 1},
		{Lo: 0x301d, Hi: 0x301d, Stride: 1},
		{Lo: 0x301e, Hi: 0x301f, Stride: 1},
		{Lo: 0x3020, Hi: 0x3020, Stride: 1},
		{Lo: 0x3021, Hi: 0x3029, Stride: 1},
		{Lo: 0x302a, Hi: 0x302d, Stride: 1},
		{Lo: 0x302e, Hi: 0x302f, Stride: 1},
		{Lo: 0x3030, Hi: 0x3030, Stride: 1},
		{Lo: 0x3031, Hi: 0x3035, Stride: 1},
		{Lo: 0x3036, Hi: 0x3037, Stride: 1},
		{Lo: 0x3038, Hi: 0x303a, Stride: 1},
		{Lo: 0x303b, Hi: 0x303b, Stride: 1},
		{Lo: 0x303c, Hi: 0x303c, Stride: 1},
		{Lo: 0x303d, Hi: 0x303d, Stride: 1},
		{Lo: 0x303e, Hi: 0x303e, Stride: 1},
		{Lo: 0x3041, Hi: 0x3096, Stride: 1},
		{Lo: 0x3099, Hi: 0x309a, Stride: 1},
		{Lo: 0x309b, Hi: 0x309c, Stride: 1},
		{Lo: 0x309d, Hi: 0x309e, Stride: 1},
		{Lo: 0x309f, Hi: 0x309f, Stride: 1},
		{Lo: 0x30a0, Hi: 0x30a0, Stride: 1},
		{Lo: 0x30a1, Hi: 0x30fa, Stride: 1},
		{Lo: 0x30fb, Hi: 0x30fb, Stride: 1},
		{Lo: 0x30fc, Hi: 0x30fe, Stride: 1},
		{Lo: 0x30ff, Hi: 0x30ff, Stride: 1},
		{Lo: 0x3105, Hi: 0x312f, Stride: 1},
		{Lo: 0x3131, Hi: 0x318e, Stride: 1},
		{Lo: 0x3190, Hi: 0x3191, Stride: 1},
		{Lo: 0x3192, Hi: 0x3195, Stride: 1},
		{Lo: 0x3196, Hi: 0x319f, Stride: 1},
		{Lo: 0x31a0, Hi: 0x31ba, Stride: 1},
		{Lo: 0x31c0, Hi: 0x31e3, Stride: 1},
		{Lo: 0x31f0, Hi: 0x31ff, Stride: 1},
		{Lo: 0x3200, Hi: 0x321e, Stride: 1},
		{Lo: 0x3220, Hi: 0x3229, Stride: 1},
		{Lo: 0x322a, Hi: 0x3247, Stride: 1},
		{Lo: 0x3250, Hi: 0x3250, Stride: 1},
		{Lo: 0x3251, Hi: 0x325f, Stride: 1},
		{Lo: 0x3260, Hi: 0x327f, Stride: 1},
		{Lo: 0x3280, Hi: 0x3289, Stride: 1},
		{Lo: 0x328a, Hi: 0x32b0, Stride: 1},
		{Lo: 0x32b1, Hi: 0x32bf, Stride: 1},
		{Lo: 0x32c0, Hi: 0x32fe, Stride: 1},
		{Lo: 0x3300, Hi: 0x33ff, Stride: 1},
		{Lo: 0x3400, Hi: 0x4db5, Stride: 1},
		{Lo: 0x4db6, Hi: 0x4dbf, Stride: 1},
		{Lo: 0x4e00, Hi: 0x9fef, Stride: 1},
		{Lo: 0x9ff0, Hi: 0x9fff, Stride: 1},
		{Lo: 0xa000, Hi: 0xa014, Stride: 1},
		{Lo: 0xa015, Hi: 0xa015, Stride: 1},
		{Lo: 0xa016, Hi: 0xa48c, Stride: 1},
		{Lo: 0xa490, Hi: 0xa4c6, Stride: 1},
		{Lo: 0xa960, Hi: 0xa97c, Stride: 1},
		{Lo: 0xac00, Hi: 0xd7a3, Stride: 1},
		{Lo: 0xf900, Hi: 0xfa6d, Stride: 1},
		{Lo: 0xfa6e, Hi: 0xfa6f, Stride: 1},
		{Lo: 0xfa70, Hi: 0xfad9, Stride: 1},
		{Lo: 0xfada, Hi: 0xfaff, Stride: 1},
		{Lo: 0xfe10, Hi: 0xfe16, Stride: 1},
		{Lo: 0xfe17, Hi: 0xfe17, Stride: 1},
		{Lo: 0xfe18, Hi: 0xfe18, Stride: 1},
		{Lo: 0xfe19, Hi: 0xfe19, Stride: 1},
		{Lo: 0xfe30, Hi: 0xfe30, Stride: 1},
		{Lo: 0xfe31, Hi: 0xfe32, Stride: 1},
		{Lo: 0xfe33, Hi: 0xfe34, Stride: 1},
		{Lo: 0xfe35, Hi: 0xfe35, Stride: 1},
		{Lo: 0xfe36, Hi: 0xfe36, Stride: 1},
		{Lo: 0xfe37, Hi: 0xfe37, Stride: 1},
		{Lo: 0xfe38, Hi: 0xfe38, Stride: 1},
		{Lo: 0xfe39, Hi: 0xfe39, Stride: 1},
		{Lo: 0xfe3a, Hi: 0xfe3a, Stride: 1},
		{Lo: 0xfe3b, Hi: 0xfe3b, Stride: 1},
		{Lo: 0xfe3c, Hi: 0xfe3c, Stride: 1},
		{Lo: 0xfe3d, Hi: 0xfe3d, Stride: 1},
		{Lo: 0xfe3e, Hi: 0xfe3e, Stride: 1},
		{Lo: 0xfe3f, Hi: 0xfe3f, Stride: 1},
		{Lo: 0xfe40, Hi: 0xfe40, Stride: 1},
		{Lo: 0xfe41, Hi: 0xfe41, Stride: 1},
		{Lo: 0xfe42, Hi: 0xfe42, Stride: 1},
		{Lo: 0xfe43, Hi: 0xfe43, Stride: 1},
		{Lo: 0xfe44, Hi: 0xfe44, Stride: 1},
		{Lo: 0xfe45, Hi: 0xfe46, Stride: 1},
		{Lo: 0xfe47, Hi: 0xfe47, Stride: 1},
		{Lo: 0xfe48, Hi: 0xfe48, Stride: 1},
		{Lo: 0xfe49, Hi: 0xfe4c, Stride: 1},
		{Lo: 0xfe4d, Hi: 0xfe4f, Stride: 1},
		{Lo: 0xfe50, Hi: 0xfe52, Stride: 1},
		{Lo: 0xfe54, Hi: 0xfe57, Stride: 1},
		{Lo: 0xfe58, Hi: 0xfe58, Stride: 1},
		{Lo: 0xfe59, Hi: 0xfe59, Stride: 1},
		{Lo: 0xfe5a, Hi: 0xfe5a, Stride: 1},
		{Lo: 0xfe5b, Hi: 0xfe5b, Stride: 1},
		{Lo: 0xfe5c, Hi: 0xfe5c, Stride: 1},
		{Lo: 0xfe5d, Hi: 0xfe5d, Stride: 1},
		{Lo: 0xfe5e, Hi: 0xfe5e, Stride: 1},
		{Lo: 0xfe5f, Hi: 0xfe61, Stride: 1},
		{Lo: 0xfe62, Hi: 0xfe62, Stride: 1},
		{Lo: 0xfe63, Hi: 0xfe63, Stride: 1},
		{Lo: 0xfe64, Hi: 0xfe66, Stride: 1},
		{Lo: 0xfe68, Hi: 0xfe68, Stride: 1},
		{Lo: 0xfe69, Hi: 0xfe69, Stride: 1},
		{Lo: 0xfe6a, Hi: 0xfe6b, Stride: 1},
		{Lo: 0xff01, Hi: 0xff03, Stride: 1},
		{Lo: 0xff04, Hi: 0xff04, Stride: 1},
		{Lo: 0xff05, Hi: 0xff07, Stride: 1},
		{Lo: 0xff08, Hi: 0xff08, Stride: 1},
		{Lo: 0xff09, Hi: 0xff09, Stride: 1},
		{Lo: 0xff0a, Hi: 0xff0a, Stride: 1},
		{Lo: 0xff0b, Hi: 0xff0b, Stride: 1},
		{Lo: 0xff0c, Hi: 0xff0c, Stride: 1},
		{Lo: 0xff0d, Hi: 0xff0d, Stride: 1},
		{Lo: 0xff0e, Hi: 0xff0f, Stride: 1},
		{Lo: 0xff10, Hi: 0xff19, Stride: 1},
		{Lo: 0xff1a, Hi: 0xff1b, Stride: 1},
		{Lo: 0xff1c, Hi: 0xff1e, Stride: 1},
		{Lo: 0xff1f, Hi: 0xff20, Stride: 1},
		{Lo: 0xff21, Hi: 0xff3a, Stride: 1},
		{Lo: 0xff3b, Hi: 0xff3b, Stride: 1},
		{Lo: 0xff3c, Hi: 0xff3c, Stride: 1},
		{Lo: 0xff3d, Hi: 0xff3d, Stride: 1},
		{Lo: 0xff3e, Hi: 0xff3e, Stride: 1},
		{Lo: 0xff3f, Hi: 0xff3f, Stride: 1},
		{Lo: 0xff40, Hi: 0xff40, Stride: 1},
		{Lo: 0xff41, Hi: 0xff5a, Stride: 1},
		{Lo: 0xff5b, Hi: 0xff5b, Stride: 1},
		{Lo: 0xff5c, Hi: 0xff5c, Stride: 1},
		{Lo: 0xff5d, Hi: 0xff5d, Stride: 1},
		{Lo: 0xff5e, Hi: 0xff5e, Stride: 1},
		{Lo: 0xff5f, Hi: 0xff5f, Stride: 1},
		{Lo: 0xff60, Hi: 0xff60, Stride: 1},
		{Lo: 0xffe0, Hi: 0xffe1, Stride: 1},
		{Lo: 0xffe2, Hi: 0xffe2, Stride: 1},
		{Lo: 0xffe3, Hi: 0xffe3, Stride: 1},
		{Lo: 0xffe4, Hi: 0xffe4, Stride: 1},
		{Lo: 0xffe5, Hi: 0xffe6, Stride: 1},
	},
	R32: []unicode.Range32{
		{Lo: 0x16fe0, Hi: 0x16fe1, Stride: 1},
		{Lo: 0x17000, Hi: 0x187f1, Stride: 1},
		{Lo: 0x18800, Hi: 0x18af2, Stride: 1},
		{Lo: 0x1b000, Hi: 0x1b0ff, Stride: 1},
		{Lo: 0x1b100, Hi: 0x1b11e, Stride: 1},
		{Lo: 0x1b170, Hi: 0x1b2fb, Stride: 1},
		{Lo: 0x1f004, Hi: 0x1f004, Stride: 1},
		{Lo: 0x1f0cf, Hi: 0x1f0cf, Stride: 1},
		{Lo: 0x1f18e, Hi: 0x1f18e, Stride: 1},
		{Lo: 0x1f191, Hi: 0x1f19a, Stride: 1},
		{Lo: 0x1f200, Hi: 0x1f202, Stride: 1},
		{Lo: 0x1f210, Hi: 0x1f23b, Stride: 1},
		{Lo: 0x1f240, Hi: 0x1f248, Stride: 1},
		{Lo: 0x1f250, Hi: 0x1f251, Stride: 1},
		{Lo: 0x1f260, Hi: 0x1f265, Stride: 1},
		{Lo: 0x1f300, Hi: 0x1f320, Stride: 1},
		{Lo: 0x1f32d, Hi: 0x1f335, Stride: 1},
		{Lo: 0x1f337, Hi: 0x1f37c, Stride: 1},
		{Lo: 0x1f37e, Hi: 0x1f393, Stride: 1},
		{Lo: 0x1f3a0, Hi: 0x1f3ca, Stride: 1},
		{Lo: 0x1f3cf, Hi: 0x1f3d3, Stride: 1},
		{Lo: 0x1f3e0, Hi: 0x1f3f0, Stride: 1},
		{Lo: 0x1f3f4, Hi: 0x1f3f4, Stride: 1},
		{Lo: 0x1f3f8, Hi: 0x1f3fa, Stride: 1},
		{Lo: 0x1f3fb, Hi: 0x1f3ff, Stride: 1},
		{Lo: 0x1f400, Hi: 0x1f43e, Stride: 1},
		{Lo: 0x1f440, Hi: 0x1f440, Stride: 1},
		{Lo: 0x1f442, Hi: 0x1f4fc, Stride: 1},
		{Lo: 0x1f4ff, Hi: 0x1f53d, Stride: 1},
		{Lo: 0x1f54b, Hi: 0x1f54e, Stride: 1},
		{Lo: 0x1f550, Hi: 0x1f567, Stride: 1},
		{Lo: 0x1f57a, Hi: 0x1f57a, Stride: 1},
		{Lo: 0x1f595, Hi: 0x1f596, Stride: 1},
		{Lo: 0x1f5a4, Hi: 0x1f5a4, Stride: 1},
		{Lo: 0x1f5fb, Hi: 0x1f5ff, Stride: 1},
		{Lo: 0x1f600, Hi: 0x1f64f, Stride: 1},
		{Lo: 0x1f680, Hi: 0x1f6c5, Stride: 1},
		{Lo: 0x1f6cc, Hi: 0x1f6cc, Stride: 1},
		{Lo: 0x1f6d0, Hi: 0x1f6d2, Stride: 1},
		{Lo: 0x1f6eb, Hi: 0x1f6ec, Stride: 1},
		{Lo: 0x1f6f4, Hi: 0x1f6f9, Stride: 1},
		{Lo: 0x1f910, Hi: 0x1f93e, Stride: 1},
		{Lo: 0x1f940, Hi: 0x1f970, Stride: 1},
		{Lo: 0x1f973, Hi: 0x1f976, Stride: 1},
		{Lo: 0x1f97a, Hi: 0x1f97a, Stride: 1},
		{Lo: 0x1f97c, Hi: 0x1f9a2, Stride: 1},
		{Lo: 0x1f9b0, Hi: 0x1f9b9, Stride: 1},
		{Lo: 0x1f9c0, Hi: 0x1f9c2, Stride: 1},
		{Lo: 0x1f9d0, Hi: 0x1f9ff, Stride: 1},
		{Lo: 0x20000, Hi: 0x2a6d6, Stride: 1},
		{Lo: 0x2a6d7, Hi: 0x2a6ff, Stride: 1},
		{Lo: 0x2a700, Hi: 0x2b734, Stride: 1},
		{Lo: 0x2b735, Hi: 0x2b73f, Stride: 1},
		{Lo: 0x2b740, Hi: 0x2b81d, Stride: 1},
		{Lo: 0x2b81e, Hi: 0x2b81f, Stride: 1},
		{Lo: 0x2b820, Hi: 0x2cea1, Stride: 1},
		{Lo: 0x2cea2, Hi: 0x2ceaf, Stride: 1},
		{Lo: 0x2ceb0, Hi: 0x2ebe0, Stride: 1},
		{Lo: 0x2ebe1, Hi: 0x2f7ff, Stride: 1},
		{Lo: 0x2f800, Hi: 0x2fa1d, Stride: 1},
		{Lo: 0x2fa1e, Hi: 0x2fa1f, Stride: 1},
		{Lo: 0x2fa20, Hi: 0x2fffd, Stride: 1},
		{Lo: 0x30000, Hi: 0x3fffd, Stride: 1},
	},
}

View Source

var LineBreakLiteral = map[LineBreak]string{
	CR:   "CR",
	LF:   "LF",
	CRLF: "CRLF",
}

View Source

var RightToLeftTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x0590, Hi: 0x05ff, Stride: 1},
		{Lo: 0x0600, Hi: 0x06ff, Stride: 1},
		{Lo: 0x0700, Hi: 0x074f, Stride: 1},
		{Lo: 0x0750, Hi: 0x077f, Stride: 1},
		{Lo: 0x0860, Hi: 0x086f, Stride: 1},
		{Lo: 0x08a0, Hi: 0x08ff, Stride: 1},
		{Lo: 0xfb50, Hi: 0xfdff, Stride: 1},
		{Lo: 0xfe70, Hi: 0xfeff, Stride: 1},
	},
	R32: []unicode.Range32{
		{Lo: 0x10ac0, Hi: 0x10aff, Stride: 1},
		{Lo: 0x10c00, Hi: 0x10c4f, Stride: 1},
		{Lo: 0x10f00, Hi: 0x10f2f, Stride: 1},
		{Lo: 0x10f30, Hi: 0x10f6f, Stride: 1},
		{Lo: 0x1ee00, Hi: 0x1eeff, Stride: 1},
	},
}

View Source

var SJISSingleByteTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x0020, Hi: 0x007e, Stride: 1},
		{Lo: 0xff61, Hi: 0xff9f, Stride: 1},
	},
}

View Source

var ZeroWidthSpaceTable = &unicode.RangeTable{
	R16: []unicode.Range16{
		{Lo: 0x200b, Hi: 0x200b, Stride: 1},
		{Lo: 0x2060, Hi: 0x2063, Stride: 1},
		{Lo: 0xfeff, Hi: 0xfeff, Stride: 1},
	},
}

Functions ¶

func ByteSize ¶

func ByteSize(s string, encoding Encoding) int

ByteSize calculates byte size of a string.

func Decode ¶

func Decode(src []byte, enc Encoding) ([]byte, error)

Decode a string from any encoding to UTF-8.

func Encode ¶

func Encode(src []byte, enc Encoding) ([]byte, error)

Encode a string from UTF-8 to another encoding.

func GetTransformDecoder ¶

func GetTransformDecoder(r io.Reader, enc Encoding) (io.Reader, error)

GetTransformDecoder gets a reader to transform character encoding from any encoding to UTF-8.

func GetTransformEncoder ¶

func GetTransformEncoder(r io.Reader, enc Encoding) (io.Reader, error)

GetTransformEncoder gets a reader to transform character encoding from UTF-8 to another encoding.

func GetTransformWriter ¶

func GetTransformWriter(w io.Writer, enc Encoding) (io.Writer, error)

GetTransformWriter gets a writer to transform character encoding from UTF-8 to another encoding.

func IsHighSurrogate ¶

func IsHighSurrogate(r rune) bool

func IsLowSurrogate ¶

func IsLowSurrogate(r rune) bool

func IsRightToLeftLetters ¶

func IsRightToLeftLetters(s string) bool

IsRightToLeftLetters returns true if a string is Right-to-Left horizontal writing characters.

func NewUTF8MEncoder ¶

func NewUTF8MEncoder() *encoding.Encoder

func RuneByteSize ¶

func RuneByteSize(r rune, encoding Encoding) int

RuneByteSize calculates byte size of a character.

func RuneWidth ¶

func RuneWidth(r rune, eastAsianEncoding bool, countDiacriticalSign bool, countFormatCode bool) int

RuneWidth calculates character width to be displayed.

func Width ¶

func Width(s string, eastAsianEncoding bool, countDiacriticalSign bool, countFormatCode bool) int

Width calculates string width to be displayed.

Types ¶

type Encoding ¶

type Encoding uint8

const (
	AUTO Encoding = iota
	UTF8
	UTF8M
	UTF16
	UTF16BEM
	UTF16LEM
	UTF16BE
	UTF16LE
	SJIS
)

func DetectEncoding ¶

func DetectEncoding(r io.ReadSeeker) (detected Encoding, err error)

DetectEncoding Detects character encoding

func DetectInSpecifiedEncoding ¶

func DetectInSpecifiedEncoding(r io.ReadSeeker, enc Encoding) (detected Encoding, err error)

func InferEncoding ¶

func InferEncoding(b []byte, eof bool) (Encoding, error)

func ParseEncoding ¶

func ParseEncoding(s string) (Encoding, error)

func (Encoding) String ¶

func (e Encoding) String() string

type FieldAlignment ¶

type FieldAlignment int

const (
	NotAligned FieldAlignment = iota
	Centering
	RightAligned
	LeftAligned
)

type LineBreak ¶

type LineBreak string

const (
	CR   LineBreak = "\r"
	LF   LineBreak = "\n"
	CRLF LineBreak = "\r\n"
)

func ParseLineBreak ¶

func ParseLineBreak(s string) (LineBreak, error)

func (LineBreak) String ¶

func (lb LineBreak) String() string

func (LineBreak) Value ¶

func (lb LineBreak) Value() string

type RawText ¶

type RawText []byte

type UTF8MEncoder ¶

type UTF8MEncoder struct {
	// contains filtered or unexported fields
}

func (*UTF8MEncoder) Reset ¶

func (u *UTF8MEncoder) Reset()

func (*UTF8MEncoder) Transform ¶

func (u *UTF8MEncoder) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error)

Source Files ¶

View all Source files

Directories ¶

Path	Synopsis
color Package color is a Go library to render ANSI escape color sequences.	Package color is a Go library to render ANSI escape color sequences.
csv Package csv is a Go library to read and write CSV format.	Package csv is a Go library to read and write CSV format.
fixedlen Package fixedlen is a Go library to read and write Fixed-Length format.	Package fixedlen is a Go library to read and write Fixed-Length format.
json Package json is a Go library to read and write JSON format.	Package json is a Go library to read and write JSON format.
jsonl Package jsonl is a Go library to read and write JSON Lines format.	Package jsonl is a Go library to read and write JSON Lines format.
ltsv Package ltsv is a Go library to read and write LTSV format.	Package ltsv is a Go library to read and write LTSV format.
table Package table is a Go library to write text tables.	Package table is a Go library to write text tables.

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL