lex

README

goulash/lex

Package lex helps you implement your own lexer and parser.

The code and API are strongly influenced by the lexer found in the Go standard library package text/template/parse. For help on how to use this package, see also the package pre.

For more information, see the documentation! :-) This package is licensed under the MIT license.

Documentation

Overview

Package lex provides a base lexer to help you implement your own.

Feel free to copy this into your library (according to the license) as well as use it as a library. Both methods are possible.

Code and API based off standard library text/template/parse.

Index

• Constants
• Variables
• func IsAlphaNumeric(r rune) bool
• func IsEndline(r rune) bool
• func IsQuote(r rune) bool
• func IsSpace(r rune) bool
• type Lexer
  • func Lex(name, input string, sf StateFn) *Lexer
  • func New(name, input string) *Lexer
  • func (l *Lexer) Accept(valid string) bool
  • func (l *Lexer) AcceptBut(invalid string) bool
  • func (l *Lexer) AcceptButRun(invalid string) int
  • func (l *Lexer) AcceptFunc(f func(r rune) bool) bool
  • func (l *Lexer) AcceptFuncRun(f func(r rune) bool) int
  • func (l *Lexer) AcceptRun(valid string) int
  • func (l *Lexer) Backup()
  • func (l *Lexer) ColumnNumber() int
  • func (l *Lexer) Consume(s string) bool
  • func (l *Lexer) Dec(n int)
  • func (l *Lexer) Drain()
  • func (l *Lexer) Emit(t Type)
  • func (l *Lexer) Errorf(format string, args ...interface{}) StateFn
  • func (l *Lexer) HasPrefix(s string) bool
  • func (l *Lexer) HasPrefixAfter(after int, s string) bool
  • func (l *Lexer) Ignore()
  • func (l *Lexer) Inc(n int)
  • func (l *Lexer) Input(n int) string
  • func (l *Lexer) Len() int
  • func (l *Lexer) LineNumber() int
  • func (l *Lexer) Name() string
  • func (l *Lexer) Next() rune
  • func (l *Lexer) NextToken() Token
  • func (l *Lexer) Peek() rune
  • func (l *Lexer) Pos() int
  • func (l *Lexer) Run(fn StateFn)
  • func (l *Lexer) Value() string
• type Reader
  • func NewReader(l *Lexer) *Reader
  • func (r *Reader) Backup(t Token)
  • func (r *Reader) Expect(types ...Type) ([]Token, bool)
  • func (r *Reader) Next() Token
  • func (r *Reader) Peek() Token
  • func (r *Reader) PosInfo() (name string, line, col int)
• type StateFn
• type Token
• type Type

Constants

const EOF = -1

EOF is returned by Lexer.Next upon reaching end-of-file.

Variables

var (
	Space   = " \t"
	Endline = "\r\n"
	Quote   = "\"'`"
)

Functions

func IsAlphaNumeric

func IsAlphaNumeric(r rune) bool

func IsEndline

func IsEndline(r rune) bool

func IsQuote

func IsQuote(r rune) bool

func IsSpace

func IsSpace(r rune) bool

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

func Lex

func Lex(name, input string, sf StateFn) *Lexer

Lex creates a new Lexer and starts running it with sf.

func New

func New(name, input string) *Lexer

New creates a new Lexer and returns it.

Before calling NextToken, it should be run in a separate goroutine:

l := lex.New(name, input)
go l.Run(sf)
...
t := l.NextToken()

func (*Lexer) Accept

func (l *Lexer) Accept(valid string) bool

Accept consumes the next rune if it is from the valid set.

func (*Lexer) AcceptBut

func (l *Lexer) AcceptBut(invalid string) bool

AcceptBut consumes a rune if it is not from the invalid set.

func (*Lexer) AcceptButRun

func (l *Lexer) AcceptButRun(invalid string) int

AcceptButRun consumes runes as long as they are not in the invalid set. The number of bytes advanced is returned.

func (*Lexer) AcceptFunc

func (l *Lexer) AcceptFunc(f func(r rune) bool) bool

AcceptFunc consumes the next rune if f returns true.

func (*Lexer) AcceptFuncRun

func (l *Lexer) AcceptFuncRun(f func(r rune) bool) int

AcceptFunc consumes a run of runes as long as f returns true. The number of bytes advanced is returned.

func (*Lexer) AcceptRun

func (l *Lexer) AcceptRun(valid string) int

AcceptRun consumes a run of runes from the valid set. The number of bytes advanced is returned.

func (*Lexer) Backup

func (l *Lexer) Backup()

Backup steps back one rune. Can only be called once per call of Next.

func (*Lexer) ColumnNumber

func (l *Lexer) ColumnNumber() int

ColumnNumber reports the column of the last token returned by NextToken.

func (*Lexer) Consume

func (l *Lexer) Consume(s string) bool

Consume tries to consume exactly the string s.

func (*Lexer) Dec

func (l *Lexer) Dec(n int)

Dec decrements the position by n.

func (*Lexer) Drain

func (l *Lexer) Drain()

Drain drains the output so the lexing goroutine will exit. Called by the parser, not in the lexing goroutine.

func (*Lexer) Emit

func (l *Lexer) Emit(t Type)

Emit passes a token back to the client.

func (*Lexer) Errorf

func (l *Lexer) Errorf(format string, args ...interface{}) StateFn

Errorf returns an error token and terminates the scan by passing back a nil pointer that will be the next state, terminating l.NextToken.

func (*Lexer) HasPrefix

func (l *Lexer) HasPrefix(s string) bool

HasPrefix returns true if the input from the current position has the prefix s. It does not consume the prefix.

func (*Lexer) HasPrefixAfter

func (l *Lexer) HasPrefixAfter(after int, s string) bool

HasPrefixAfter returns true if the input from the current position plus after bytes has the prefix s. It does not consume the prefix.

func (*Lexer) Ignore

func (l *Lexer) Ignore()

Ignore skips over the pending input before this point.

func (*Lexer) Inc

func (l *Lexer) Inc(n int)

Inc increments the position by n.

func (*Lexer) Input

func (l *Lexer) Input(n int) string

Input returns a slice of the current position plus n.

func (*Lexer) Len

func (l *Lexer) Len() int

Len returns the size of the current read token.

func (*Lexer) LineNumber

func (l *Lexer) LineNumber() int

LineNumber reports the line of the last token returned by NextToken.

func (*Lexer) Name

func (l *Lexer) Name() string

Name returns the name of the input.

func (*Lexer) Next

func (l *Lexer) Next() rune

Next returns the next rune in the input. If there is no more input left to read, EOF is returned.

func (*Lexer) NextToken

func (l *Lexer) NextToken() Token

NextToken returns the next token from the input. Called by the parser, not in the lexing goroutine.

Note: if l.Run has not been called, NextToken will block.

func (*Lexer) Peek

func (l *Lexer) Peek() rune

Peek returns but does not consume the next rune in the input.

func (*Lexer) Pos

func (l *Lexer) Pos() int

Pos returns the current position in the input.

func (*Lexer) Run

func (l *Lexer) Run(fn StateFn)

Run starts the lexer with the given StateFn. After receiving a nil StateFn, it closes the tokens channel.

func (*Lexer) Value

func (l *Lexer) Value() string

Value returns the current token value, essentially the part of input from l.base to l.pos.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

func NewReader

func NewReader(l *Lexer) *Reader

func (*Reader) Backup

func (r *Reader) Backup(t Token)

func (*Reader) Expect

func (r *Reader) Expect(types ...Type) ([]Token, bool)

Expect reads the expected tokens and returns them in a slice. If a token has an unexpected type, it is the last token in the slice and false is returned. The length of the returned slice is the number of tokens read, the capacity is the expected number of tokens to read.

func (*Reader) Next

func (r *Reader) Next() Token

func (*Reader) Peek

func (r *Reader) Peek() Token

func (*Reader) PosInfo

func (r *Reader) PosInfo() (name string, line, col int)

type StateFn

type StateFn func(*Lexer) StateFn

type Token

type Token struct {
	Type
	Pos   int
	Value string
}

type Type

type Type int

A Type is the type of a token.

In this package, only several types are predefined. The rest you can set yourself like so:

const (
    TypeSpace = (1+lex.TypeEOF)+iota // continue where lex left off
    TypeNumber
    TypeIdent
    ...
)

const (
	TypeError Type = iota // string is error text
	TypeEOF               // end-of-file, last reserved type
)