tokenizer

Documentation

Index

• Constants
• Variables
• type Lexeme
• type Token
  • func (t *Token) Inspect() string
• type Tokenizer
  • func MakeTokenizer(src []byte) *Tokenizer
  • func (t *Tokenizer) Peek() *Token
  • func (t *Tokenizer) Pop() *Token

Constants

const (
	LPAREN = iota
	RPAREN
	LBRACE
	RBRACE
	COMMA
	DOT
	EQUAL
	PLUS
	STRING
	REGEXP
	POS
	GVAR
	LVAR
	KWD
	ID
	NAMESPACE
	OPEN
	FUNC
	TYPE
	PATH
	IMPORT
	OPTIONAL
	READ
	EOF
	ERROR
	NUM_LEXEMES
)

Variables

var LexemeName [NUM_LEXEMES]string

Types

type Lexeme

type Lexeme int

Type tags so we know what kind of token we have

type Token

type Token struct {
	Lexeme
	Value      string
	ExtraValue string
	LineNumber int32
}

A token has a type (aka lexeme), a value, and a line number

func (*Token) Inspect

func (t *Token) Inspect() string

type Tokenizer

type Tokenizer struct {
	Source     []byte
	LineNumber int32
	Lookahead  *Token
	// contains filtered or unexported fields
}

Represent a tokenizer with a struct containing the remaining source text and the line number. Easier than using a stateless tokenizing function that returns them as extra values and requires the parser to keep track of them.

func MakeTokenizer

func MakeTokenizer(src []byte) *Tokenizer

func (*Tokenizer) Peek

func (t *Tokenizer) Peek() *Token

func (*Tokenizer) Pop

func (t *Tokenizer) Pop() *Token