tokenizer

Documentation

Overview

Package tokenizer implements a rudimentary tokens parser of buffered io.Reader while respecting quotes and parenthesis boundaries.

Example

tk := tokenizer.NewFromString("a, b, (c, d)")
result, _ := tk.ScanAll() // ["a", "b", "(c, d)"]

Index

• Variables
• type Tokenizer
  • func New(r io.Reader) *Tokenizer
  • func NewFromBytes(b []byte) *Tokenizer
  • func NewFromString(str string) *Tokenizer
  • func (s *Tokenizer) KeepSeparator(state bool)
  • func (s *Tokenizer) Scan() (string, error)
  • func (s *Tokenizer) ScanAll() ([]string, error)
  • func (s *Tokenizer) Separators(separators ...rune)

Variables

var DefaultSeparators = []rune{','}

DefaultSeparators is a list with the default token separator characters.

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer defines a struct that parses a reader into tokens while respecting quotes and parenthesis boundaries.

func New

func New(r io.Reader) *Tokenizer

New creates new Tokenizer from the provided reader with DefaultSeparators.

func NewFromBytes

func NewFromBytes(b []byte) *Tokenizer

NewFromBytes creates new Tokenizer from the provided bytes slice.

func NewFromString

func NewFromString(str string) *Tokenizer

NewFromString creates new Tokenizer from the provided string.

func (*Tokenizer) KeepSeparator

func (s *Tokenizer) KeepSeparator(state bool)

KeepSeparator defines whether to keep the separator rune as part of the token (default to false).

func (*Tokenizer) Scan

func (s *Tokenizer) Scan() (string, error)

Scan reads and returns the next available token from the Tokenizer's buffer (trimmed).

Returns io.EOF error when there are no more tokens to scan.

func (*Tokenizer) ScanAll

func (s *Tokenizer) ScanAll() ([]string, error)

ScanAll reads the entire Tokenizer's buffer and return all found tokens.

func (*Tokenizer) Separators

func (s *Tokenizer) Separators(separators ...rune)

Separators defines the provided separatos of the current Tokenizer.