lexer

README

Generic Go lexer

A very simple state-based lexer.

import "src.userspace.com.au/lexer"

// Define the tokens for the lexer.
const (
	_ lexer.TokenType = iota
	tBREStart
	tBREEnd
	tRangeStart
	tRangeDash
	tRangeEnd
	tCharacter
	tClass
	tNot
)

// Define states returning a StateFunc.
func startState(l *lexer.Lexer) lexer.StateFunc {
	l.SkipWhitespace()
	r := l.Next()
	if r != '[' {
		return l.Error("expecting [")
	}
	l.Emit(tBREStart)
	return beFirstState
}

For a complete (but simple) example see the code in a bracket expression generator.

Documentation

Index

• Constants
• type Lexer
  • func New(src string, start StateFunc) *Lexer
  • func (l *Lexer) Accept(valid string) bool
  • func (l *Lexer) AcceptRun(valid string) (n int)
  • func (l *Lexer) Backup()
  • func (l *Lexer) Current() string
  • func (l *Lexer) Emit(t TokenType)
  • func (l *Lexer) Error(format string, args ...interface{}) StateFunc
  • func (l *Lexer) Ignore()
  • func (l *Lexer) Next() rune
  • func (l *Lexer) NextToken() (*Token, bool)
  • func (l *Lexer) Peek() rune
  • func (l *Lexer) SkipWhitespace()
  • func (l *Lexer) Start()
  • func (l *Lexer) StartSync()
  • func (l *Lexer) Tokens() <-chan Token
• type StateFunc
• type Token
  • func (t Token) String() string
• type TokenType

Constants

const (
	// EOFRune is a convenience for EOF
	EOFRune rune = -1
	// ErrorToken is returned on error
	ErrorToken TokenType = -1
	// EOFToken is return on EOF
	EOFToken TokenType = 0
)

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer represents the lexer machine.

func New

func New(src string, start StateFunc) *Lexer

New creates a returns a lexer ready to parse the given source code.

func (*Lexer) Accept

func (l *Lexer) Accept(valid string) bool

Accept receives a string containing all acceptable strings and will continue over each consecutive character in the source until a token not in the given string is encountered. This should be used to quickly pull token parts.

func (*Lexer) AcceptRun

func (l *Lexer) AcceptRun(valid string) (n int)

AcceptRun consumes a run of runes from the valid set.

func (*Lexer) Backup

func (l *Lexer) Backup()

Backup will take the last rune read (if any) and history back. Backups can occur more than once per call to Next but you can never history past the last point a token was emitted.

func (*Lexer) Current

func (l *Lexer) Current() string

Current returns the value being being analyzed at this moment.

func (*Lexer) Emit

func (l *Lexer) Emit(t TokenType)

Emit will receive a token type and push a new token with the current analyzed value into the tokens channel.

func (*Lexer) Error

func (l *Lexer) Error(format string, args ...interface{}) StateFunc

func (*Lexer) Ignore

func (l *Lexer) Ignore()

Ignore clears the history stack and then sets the current beginning position to the current position in the source which effectively ignores the section of the source being analyzed.

func (*Lexer) Next

func (l *Lexer) Next() rune

Next pulls the next rune from the Lexer and returns it, moving the position forward in the source.

func (*Lexer) NextToken

func (l *Lexer) NextToken() (*Token, bool)

NextToken returns the next token from the lexer and done

func (*Lexer) Peek

func (l *Lexer) Peek() rune

Peek performs a Next operation immediately followed by a Backup returning the peeked rune.

func (*Lexer) SkipWhitespace

func (l *Lexer) SkipWhitespace()

SkipWhitespace continues over all unicode whitespace.

func (*Lexer) Start

func (l *Lexer) Start()

Start begins executing the Lexer in an asynchronous manner (using a goroutine).

func (*Lexer) StartSync

func (l *Lexer) StartSync()

StartSync starts the lexer synchronously.

func (*Lexer) Tokens

func (l *Lexer) Tokens() <-chan Token

Tokens returns the a token channel.

type StateFunc

type StateFunc func(*Lexer) StateFunc

StateFunc captures the movement from one state to the next.

type Token

type Token struct {
	Type     TokenType
	Value    string
	Position int
	Line     int
}

Token is returned by the lexer.

func (Token) String

func (t Token) String() string

String implements Stringer

type TokenType

type TokenType int

TokenType identifies the tokens emitted.