rplex

README

# rplex

[![Build Status](https://travis-ci.org/tomnomnom/rplex.svg?branch=master)](https://travis-ci.org/tomnomnom/rplex)

rplex is a simple general purpose lexer library inspired by the talk [Lexical Scanning in Go](https://www.youtube.com/watch?v=HxaD_trXwRE) by Rob Pike.

At this stage it's just a toy.

Documentation

Index

• type LexFn
• type Lexer
  • func New(text string) *Lexer
  • func (l *Lexer) Accept(valid string) bool
  • func (l *Lexer) AcceptFunc(fn RuneCheck)
  • func (l *Lexer) AcceptRun(valid string)
  • func (l *Lexer) AcceptRunFunc(fn RuneCheck)
  • func (l *Lexer) AcceptUntil(delims string)
  • func (l *Lexer) AcceptUntilUnescaped(delims string)
  • func (l *Lexer) Backup()
  • func (l *Lexer) Emit(t Token)
  • func (l *Lexer) Ignore()
  • func (l *Lexer) Next() rune
  • func (l *Lexer) Peek() rune
  • func (l *Lexer) Run(initial LexFn) []Token
• type RuneCheck
• type TextToken
  • func (t *TextToken) SetText(text string)
  • func (t *TextToken) Text() string
• type Token

Types

type LexFn

type LexFn func(*Lexer) LexFn

A LexFn does the meat of the work. It accepts a pointer to a Lexer, manipulates its state in some way, e.g. accepts runes and emits tokens, and then returns a new LexFn to deal with the next stage of the lexing - or nil if no lexing is left to be done.

type Lexer

type Lexer struct {
	Text       string  // The raw input text
	Pos        int     // The current byte offset in the text
	Width      int     // The width of the current rune in bytes
	Cur        rune    // The rune at the current position
	Prev       rune    // The rune at the previous position
	Tokens     []Token // The tokens that have been emitted
	TokenStart int     // The starting position of the current token
}

Lexer holds the state for lexing statements

func New

func New(text string) *Lexer

New returns a new Lexer for the provided input string

func (*Lexer) Accept

func (l *Lexer) Accept(valid string) bool

Accept moves the pointer if the next rune is in the set of valid runes

func (*Lexer) AcceptFunc

func (l *Lexer) AcceptFunc(fn RuneCheck)

AcceptFunc accepts a rune if the provided runeCheck function returns true

func (*Lexer) AcceptRun

func (l *Lexer) AcceptRun(valid string)

AcceptRun continually accepts runes from the set of valid runes

func (*Lexer) AcceptRunFunc

func (l *Lexer) AcceptRunFunc(fn RuneCheck)

AcceptRunFunc continually accepts runes for as long as the runeCheck function returns true

func (*Lexer) AcceptUntil

func (l *Lexer) AcceptUntil(delims string)

AcceptUntil accepts runes until it hits a delimiter rune contained in the provided string

func (*Lexer) AcceptUntilUnescaped

func (l *Lexer) AcceptUntilUnescaped(delims string)

AcceptUntilUnescaped accepts runes until it hits a delimiter rune contained in the provided string, unless that rune was escaped with a backslash

func (*Lexer) Backup

func (l *Lexer) Backup()

Backup moves the lexer back one rune can only be used once per call of next()

func (*Lexer) Emit

func (l *Lexer) Emit(t Token)

Emit adds the current token to the token slice and moves the tokenStart pointer to the current position

func (*Lexer) Ignore

func (l *Lexer) Ignore()

Ignore skips the current token

func (*Lexer) Next

func (l *Lexer) Next() rune

Next gets the next rune in the input and updates the lexer state

func (*Lexer) Peek

func (l *Lexer) Peek() rune

Peek returns the next rune in the input without moving the internal pointer

func (*Lexer) Run

func (l *Lexer) Run(initial LexFn) []Token

Run runs the lexer and returns the lexed tokens

type RuneCheck

type RuneCheck func(rune) bool

RuneCheck is a function that determines if a rune is valid or not when using AcceptFunc or AcceptRunFunc. Some functions in the standard library, such as unicode.IsNumber() meet this interface already.

type TextToken

type TextToken struct {
	// contains filtered or unexported fields
}

TextToken is a generic token that can be easily embedded into custom token types to meet the Token interface

func (*TextToken) SetText

func (t *TextToken) SetText(text string)

SetText sets the text value of a TextToken

func (*TextToken) Text

func (t *TextToken) Text() string

Text gets the text value of a TextToken

type Token

type Token interface {
	SetText(string)
	Text() string
}

A Token is a chunk of text