README
¶
utf8reader
A simple go package that converts an io.Reader to a utf8 encoded io.Reader. It automatically detects the encoding of the input and converts it to utf8.
Usage
package main
import (
"fmt"
"bytes"
"github.com/kpym/utf8reader"
)
func main() {
// Create a reader with koi8-r encoded "Това е на български"
r := bytes.NewReader([]byte{0xF4, 0xCF, 0xD7, 0xC1, 0x20, 0xC5, 0x20, 0xCE, 0xC1, 0x20, 0xC2, 0xDF, 0xCC, 0xC7, 0xC1, 0xD2, 0xD3, 0xCB, 0xC9})
reader := utf8reader.New(r)
// Read the content of the reader
buf := make([]byte, 100)
n, err := reader.Read(buf)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf[:n]))
// Output: Това е на български
}
Documentation
License
Documentation
¶
Overview ¶
utf8reader is a package that detects the encoding of a reader and provides a new reader that converts (if necessary) the input to UTF-8 (without BOM). The unicode normalization form can be set to NFC or NFD.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WithNormalizationForm ¶
func WithNormalizationForm(nor string) option
WithNormalizationForm sets the normalization form. The normalization form can be "NFC" or "NFD". By default no normalization is done.
func WithPeekSize ¶ added in v0.3.0
func WithPeekSize(size int) option
WithPeekSize sets the number of bytes to peak. By default it peaks 4096 bytes. The peaked bytes are used to detect the encoding.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader is a reader that converts the input to UTF-8.
func New ¶
New returns a reader that converts the input to UTF-8 if it is not already encoded in UTF-8. If the encoding cannot be detected it returns buffered version of the original reader.
Source Files