stateful

Documentation

Overview

Package stateful defines a nested stateful lexer.

This lexer is based heavily on the approach used by Chroma (and Pygments).

The lexer is a state machine defined by a map of rules keyed by state. Each rule is a named regex and optional operation to apply when the rule matches.

As a convenience, any Rule starting with a lowercase letter will be elided from output.

Lexing starts in the "Root" group. Each rule is matched in order, with the first successful match producing a lexeme. If the matching rule has an associated Action it will be executed. The name of each non-root rule is prefixed with the name of its group to yield the token identifier used during matching.

A state change can be introduced with the Action `Push(state)`. `Pop()` will return to the previous state.

To reuse rules from another state, use `Include(state)`.

As a special case, regexes containing backrefs in the form \N (where N is a digit) will match the corresponding capture group from the immediate parent group. This can be used to parse, among other things, heredocs.

See the example and tests in this package for details.

Index

• type Action
  • func Pop() Action
  • func Push(state string) Action
• type ActionFunc
• type Definition
  • func New(rules Rules) (*Definition, error)
  • func NewSimple(rules []Rule) (*Definition, error)
  • func (d *Definition) Lex(r io.Reader) (lexer.Lexer, error)
  • func (d *Definition) Symbols() map[string]rune
• type Lexer
  • func (l *Lexer) Next() (lexer.Token, error)
• type Rule
  • func Include(state string) Rule
  • func Return() Rule
• type Rules
• type RulesAction

Examples

• New

Types

type Action

type Action interface {
	// contains filtered or unexported methods
}

A Action is applied when a rule matches.

func Pop

func Pop() Action

Pop to the previous state.

func Push

func Push(state string) Action

Push to the given state.

The target state will then be the set of rules used for matching until another Push or Pop is encountered.

type ActionFunc

type ActionFunc func(*Lexer, []string) error

ActionFunc is a function that is also a Action.

type Definition

type Definition struct {
	// contains filtered or unexported fields
}

Definition is the lexer.Definition.

func New

func New(rules Rules) (*Definition, error)

New constructs a new stateful lexer from rules.

Example

An example of parsing nested expressions within strings.

type Terminal struct {
	String *String `  @@`
	Ident  string  `| @Ident`
}

type Expr struct {
	Left  *Terminal `@@`
	Op    string    `( @Oper`
	Right *Terminal `  @@)?`
}

type Fragment struct {
	Escaped string `(  @Escaped`
	Expr    *Expr  ` | "${" @@ "}"`
	Text    string ` | @Char)`
}

type String struct {
	Fragments []*Fragment `"\"" @@* "\""`
}

def, err := New(interpolatedRules)
if err != nil {
	log.Fatal(err)
}
parser, err := participle.Build(&String{}, participle.Lexer(def))
if err != nil {
	log.Fatal(err)
}

actual := &String{}
err = parser.ParseString(`"hello ${user + "??"}"`, actual)
if err != nil {
	log.Fatal(err)
}
repr.Println(actual)

func NewSimple

func NewSimple(rules []Rule) (*Definition, error)

NewSimple creates a new stateful lexer with a single "Root" state.

func (*Definition) Lex

func (d *Definition) Lex(r io.Reader) (lexer.Lexer, error)

func (*Definition) Symbols

func (d *Definition) Symbols() map[string]rune

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer implementation.

func (*Lexer) Next

func (l *Lexer) Next() (lexer.Token, error)

type Rule

type Rule struct {
	Name    string
	Pattern string
	Action  Action
}

A Rule matching input and possibly changing state.

func Include

func Include(state string) Rule

Include rules from another state in this one.

func Return

func Return() Rule

Return to the parent state.

Useful as the last rule in a sub-state.

type Rules

type Rules map[string][]Rule

Rules grouped by name.

type RulesAction

type RulesAction interface {
	// contains filtered or unexported methods
}

RulesAction is an optional interface that Actions can implement.

It is applied during rule construction to mutate the rule map.