tokenizer

Documentation

Index

• func IsLeadingIdentifierCharacter(ch rune) bool
• type Tokenizer
  • func New(src string) *Tokenizer
  • func (t *Tokenizer) Pos() int
  • func (t *Tokenizer) Scan() (ast.Token, string, int)

Functions

func IsLeadingIdentifierCharacter

func IsLeadingIdentifierCharacter(ch rune) bool

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer represents a lexical scanner.

This implements its own internal rune buffer in order to support both read() and unread(). It's not possible to use bufio's UnreadRune() because this can only unread a single rune, and Peek() clears the rune buffer.

func New

func New(src string) *Tokenizer

New returns a new lexer instance.

func (*Tokenizer) Pos

func (t *Tokenizer) Pos() int

func (*Tokenizer) Scan

func (t *Tokenizer) Scan() (ast.Token, string, int)

Scan returns the next token and literal value.