lexer

Documentation

Overview

Package lexer tokenizes MX Script source code into a stream of tokens that the parser can consume. It tracks line and column numbers so error messages can point at the exact location of a problem.

Index

• type Lexer
  • func New(src string) *Lexer
  • func (l *Lexer) Tokenize() ([]Token, error)
• type Token
  • func (t Token) String() string
• type TokenType
  • func (t TokenType) String() string

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

func New

func New(src string) *Lexer

func (*Lexer) Tokenize

func (l *Lexer) Tokenize() ([]Token, error)

Tokenize runs the lexer over the source and returns the full token stream. It always appends a trailing TokenEOF.

type Token

type Token struct {
	Type   TokenType
	Lexeme string
	Line   int
	Col    int
}

func (Token) String

func (t Token) String() string

type TokenType

type TokenType int

const (
	TokenEOF TokenType = iota
	TokenIllegal

	TokenIdent
	TokenNumber
	TokenString

	TokenLet
	TokenFn
	TokenReturn
	TokenIf
	TokenElse
	TokenLoop
	TokenAs
	TokenRoute
	TokenServer
	TokenMiddleware
	TokenUse
	TokenTry
	TokenCatch
	TokenTrue
	TokenFalse
	TokenNull
	TokenImport
	TokenExport
	TokenWhile
	TokenBreak
	TokenContinue
	TokenStatic
	TokenMatch
	TokenFatArrow

	TokenLBrace
	TokenRBrace
	TokenLParen
	TokenRParen
	TokenLBracket
	TokenRBracket
	TokenColon
	TokenComma
	TokenDot
	TokenSemicolon
	TokenAssign
	TokenEq
	TokenNotEq
	TokenLT
	TokenGT
	TokenLTEq
	TokenGTEq
	TokenPlus
	TokenMinus
	TokenStar
	TokenSlash
	TokenPercent
	TokenBang
	TokenAnd
	TokenOr
	TokenSpread
	TokenQuestionDot  // ?.
	TokenNullCoalesce // ??
)

func (TokenType) String

func (t TokenType) String() string