lexer

Documentation

Index

• Constants
• Variables
• func IsAlnum(r rune) bool
• func IsNumeric(r rune) bool
• func StringPredicate(valid string) func(rune) bool
• type L
  • func New(name, src string) *L
  • func (l *L) AcceptOnce(pred func(rune) bool) bool
  • func (l *L) AcceptWhile(pred func(rune) bool) bool
  • func (l *L) Current() string
  • func (l *L) Emit(t tokens.TokenType)
  • func (l *L) Err() error
  • func (l *L) Errorf(format string, args ...any) lexingState
  • func (l *L) Next() rune
  • func (l *L) NextToken() tokens.Token
  • func (l *L) Peek() rune
  • func (l *L) Rewind()
  • func (l *L) Skip()
  • func (l *L) Tokens() <-chan tokens.Token

Constants

const EOF rune = -1

Variables

var ErrUnexpectedToken = errors.New("unexpected token")

Functions

func IsAlnum

func IsAlnum(r rune) bool

func IsNumeric

func IsNumeric(r rune) bool

func StringPredicate

func StringPredicate(valid string) func(rune) bool

Types

type L

type L struct {
	// contains filtered or unexported fields
}

func New

func New(name, src string) *L

func (*L) AcceptOnce

func (l *L) AcceptOnce(pred func(rune) bool) bool

AcceptOnce accepts a single rune if the predicate is true. Returns true when a rune is accepted. Otherwise, the state is rewound and false is returned.

func (*L) AcceptWhile

func (l *L) AcceptWhile(pred func(rune) bool) bool

AcceptWhile accepts runes while the predicate is true. Returns true when at least one rune is accepted. Otherwise, the state is rewound and returns false.

func (*L) Current

func (l *L) Current() string

Current returns the current token.

func (*L) Emit

func (l *L) Emit(t tokens.TokenType)

func (*L) Err

func (l *L) Err() error

func (*L) Errorf

func (l *L) Errorf(format string, args ...any) lexingState

func (*L) Next

func (l *L) Next() rune

Next returns the next rune from the source. It advances the position.

func (*L) NextToken

func (l *L) NextToken() tokens.Token

NextToken returns the next token from the lexer. It blocks until a token is available.

func (*L) Peek

func (l *L) Peek() rune

Peek returns the next rune without advancing the position.

func (*L) Rewind

func (l *L) Rewind()

Rewind unreads the last rune read from the source.

func (*L) Skip

func (l *L) Skip()

func (*L) Tokens

func (l *L) Tokens() <-chan tokens.Token

Tokens returns a channel of tokens. The channel is closed when the lexer reaches the end of the source.