lexer

Documentation

Index

• type Expression
• type Flags
• type Lexer
  • func Must(l *Lexer, err error) *Lexer
  • func New(matchers ...Matcher) (*Lexer, error)
  • func (l *Lexer) Tokenize(data string) []token.Token
• type Matcher

Types

type Expression

type Expression struct {
	Expr      string
	CloseExpr string

	Flags Flags
}

type Flags

type Flags int

const (
	NoParseInner Flags = 1 << iota
	BodyAsArg
	NoNewline
	RequireClose
)

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

* This is an implementation of the a Lexer, used to convert text into tokens * (http://en.wikipedia.org/wiki/Lexical_analysis) using the regexp package. * * Because the regexp package is implemented using a NFA * (http://en.wikipedia.org/wiki/Nondeterministic_finite_automaton), * it's very effective for this use case.

func Must

func Must(l *Lexer, err error) *Lexer

func New

func New(matchers ...Matcher) (*Lexer, error)

func (*Lexer) Tokenize

func (l *Lexer) Tokenize(data string) []token.Token

* Converts an input string into Tokens.

type Matcher

type Matcher interface {
	Name() string        // A unique name for this matcher
	Exprs() []Expression // The regular expressions that match this token

	IsValid(args *token.TokenArgs, expNum int) bool
	BuildToken(args *token.TokenArgs, expNum int) (openToken token.Token, closeToken token.Token)
}

A Matcher pairs a set of regexps and a set of tokens.