Package machines implements the lexing algorithms.
- type DFAAccepting
- type DFATrans
- type Match
- type Scanner
- type UnconsumedInput
type DFAAccepting ¶
DFAAccepting represents maps from accepting DFA states to match identifiers. These both identify which states are accepting states and which matches they belong to from the AST.
type DFATrans int
DFATrans represents a Deterministic Finite Automatons state transition table
A Match represents the positional and textual information from a match.
Equals checks two matches for equality
Scanner is a functional iterator returned by the LexerEngine. See http://hackthology.com/functional-iteration-in-go.html
func DFALexerEngine ¶
func DFALexerEngine(startState, errorState int, trans DFATrans, accepting DFAAccepting, text byte) Scanner
DFALexerEngine does the actual tokenization of the byte slice text using the DFA state machine. If the lexing process fails the Scanner will return an UnconsumedInput error.