tokenizer

Documentation

Index

• type Reader
  • func NewReader(r io.Reader) *Reader
  • func (r *Reader) Next() (string, error)
  • func (r *Reader) PeekRune() (rune, int, error)
  • func (r *Reader) Read(b []byte) (int, error)
  • func (r *Reader) ReadRune() (rune, int, error)

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader interface type is the interface for reading string with every token Reader is like bufio.Reader but has PeekRune which returns a rune without advancing pointer

func NewReader

func NewReader(r io.Reader) *Reader

NewReader creates interal reader from io.RuneReader

func (*Reader) Next

func (r *Reader) Next() (string, error)

ReadToken returns error or string as token

func (*Reader) PeekRune

func (r *Reader) PeekRune() (rune, int, error)

PeekRune returns a rune without advancing pointer

func (*Reader) Read

func (r *Reader) Read(b []byte) (int, error)

func (*Reader) ReadRune

func (r *Reader) ReadRune() (rune, int, error)

ReadRune returns a rune with advancing pointer