dsl

README

go-dsl

A small library to help you to create a Domain Specific Languager parser.

go get github.com/deslittle/go-dsl
go build github.com/deslittle/go-dsl
go test github.com/deslittle/go-dsl/mydsl

Documentation

Overview

ast.go implements an Abstract Syntax Tree for use by the DSL parser. The user tells the AST to add nodes and tokens inside the user parse function using three basic functions; p.AddNode(), p.AddToken() and p.WalkUp(). AST node types are defined by the user.

The AST is made up of Nodes (more accurately Node pointers), each of which contains a slice of Node children and a reference to it's parent. The Nodes are made available to the user so they can walk up and down the tree once it is returned from the parser.

token.go defines what a Token is and the token ID interface

parser.go implements the Parser for any DSL text. On construction it creates a new Scanner and AST and keeps a reference to them, then calls the users parser function.

When the user calls functions such as p.Expect() in their user parse function, the Parser makes calls to the Scanner to ask for a token and to the AST to create nodes and store the tokens.

scanner.go implements a Scanner for any DSL source text. When a Scanner is created it takes a bufio.Reader as the source of the text, scans a number of characters (runes) as defined by the user ScanFunc and returns a token to the Parser.

token.go defines what a Token is and the token ID interface

Index

• Constants
• func Parse(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader) (AST, []Error)
• func ParseAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader, ...) (AST, []Error)
• func ParseFile(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string) (AST, []Error)
• func ParseFileAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string, ...) (AST, []Error)
• type AST
  • func (a *AST) Inspect(fn func(*Node))
  • func (a *AST) Print()
• type Branch
• type BranchRange
• type BranchString
• type BranchToken
• type Error
  • func (e *Error) String() string
• type ErrorCode
• type ExpectRune
• type ExpectToken
• type Match
• type Node
  • func (n *Node) Print(prefix string, isTail bool)
• type NodeSet
  • func NewNodeSet(userTypes ...string) NodeSet
• type ParseFunc
• type ParseOptions
• type Parser
  • func (p *Parser) AddNode(nt string)
  • func (p *Parser) AddTokens()
  • func (p *Parser) Call(fn func(*Parser))
  • func (p *Parser) Exit() (AST, []Error)
  • func (p *Parser) Expect(expect ExpectToken)
  • func (p *Parser) GetToken() Token
  • func (p *Parser) Peek(branches []PeekToken)
  • func (p *Parser) Recover(Fn func(*Parser))
  • func (p *Parser) SkipToken()
  • func (p *Parser) WalkUp()
• type PeekToken
• type ScanFunc
• type ScanOptions
• type Scanner
  • func (s *Scanner) Call(fn func(*Scanner))
  • func (s *Scanner) Exit() Token
  • func (s *Scanner) Expect(expect ExpectRune)
  • func (s *Scanner) Match(matches []Match)
  • func (s *Scanner) SkipRune()
• type Token
• type TokenSet
  • func NewTokenSet(userIds ...string) TokenSet

Constants

const (
	NO_PREFIX indent = iota // 0
	NEWLINE
	INCREMENT
	DECREMENT
	STARTLINE
	ERROR
)

Functions

func Parse

func Parse(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader) (AST, []Error)

Parse sets up the parser, scanner and AST ready to accept input from the bufio.Reader and launches into the users entry parsing function.

The function returns the AST, Errors and the Log. The user should check len(Errors) > 0 to determine if the input was correctly formed. The log is provided to diagnose errors in the parsing/scanning logic and can be ignored once the parse/scan functions have been proven correct.

func ParseAndLog

func ParseAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, r *bufio.Reader, logfile io.Writer) (AST, []Error)

func ParseFile

func ParseFile(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string) (AST, []Error)

ParseFile sets up the bufio.Reader by accessing a file with the corresponding filename. If the file cannot be found an error will be returned at Error[0] and the AST will be nil.

If the bufio.Reader was set up correctly it will be passed to the Parse function.

func ParseFileAndLog

func ParseFileAndLog(pf ParseFunc, sf ScanFunc, ts TokenSet, ns NodeSet, inputfilename string, logfilename string) (AST, []Error)

Types

type AST

type AST struct {
	RootNode *Node `json:"root"`
	// contains filtered or unexported fields
}

RootNode is the entry point to the tree. curNode is used internally to keep track of where the next node should be added.

func (*AST) Inspect

func (a *AST) Inspect(fn func(*Node))

Inspect traverses an AST in depth-first order: It starts by calling f(node);

func (*AST) Print

func (a *AST) Print()

Prints the entire AST tree. It does so by recursively calling Print() on each node in the tree in a depth first approach.

type Branch

type Branch struct {
	Rn rune
	Fn func(*Scanner)
}

type BranchRange

type BranchRange struct {
	StartRn rune
	EndRn   rune
	Fn      func(*Scanner)
}

type BranchString

type BranchString struct {
	BranchString string
	Fn           func(*Scanner)
}

type BranchToken

type BranchToken struct {
	TokenID string
	Fn      func(*Parser)
}

ID is an interface implemented by to user so they can use their own token ID's

type Error

type Error struct {
	Code          ErrorCode
	Error         error
	LineString    string
	StartLine     int
	StartPosition int
	EndLine       int
	EndPosition   int
}

Errors contain the error text, the line and positions the error occurred on, and a string containing the input text from that line

func (*Error) String

func (e *Error) String() string

type ErrorCode

type ErrorCode int

const (
	FILE_NOT_FOUND ErrorCode = iota // 0
	COULD_NOT_CREATE_FILE
	TOKEN_EXPECTED_NOT_FOUND
	EXPECTED_TOKEN_NOT_IN_TOKENSET
	SCANNED_TOKEN_NOT_IN_TOKENSET
	RUNE_EXPECTED_NOT_FOUND
	NODE_NOT_IN_NODESET
)

type ExpectRune

type ExpectRune struct {
	Branches     []Branch
	BranchRanges []BranchRange
	Options      ScanOptions
}

type ExpectToken

type ExpectToken struct {
	Branches []BranchToken
	Options  ParseOptions
}

Used as an input to Parser.Expect()

type Match

type Match struct {
	Literal string
	ID      string
}

type Node

type Node struct {
	Type     string  `json:"type"`
	Tokens   []Token `json:"tokens"`
	Parent   *Node   `json:"-"`
	Children []*Node `json:"children"`
}

A Node can contain multiple Tokens which can be useful if the user knows how many Tokens belong to a particular Node type. Otherwise, the user should only add one token per node.

func (*Node) Print

func (n *Node) Print(prefix string, isTail bool)

Print is a recursive function that keeps track of where in the tree the node belongs to so it can print a pretty prefix. The prefix indicates how deep the node is and if it is the last node at that level.

A user can print the entire tree using AST.Print() or only print a sub-branch by calling Print() on any node in the tree.

type NodeSet

type NodeSet map[string]int

func NewNodeSet

func NewNodeSet(userTypes ...string) NodeSet

type ParseFunc

type ParseFunc func(*Parser) (AST, []Error)

The user implements the ParseFunc and passes it to the Parser in dsl.Parse()

type ParseOptions

type ParseOptions struct {
	Optional bool
	Multiple bool
	Invert   bool
	Skip     bool
}

If the Optional option is false and a match is not found, an error is returned to the parser.

If the Multiple option is set to true the parser continues to read, consume tokens and take branches until a token is read that is not matched by any of the branches. If the Multiple option is set to false, only the first branch to be matched is taken and consumed.

If the Invert option is set to true the parser consumes the token and takes a branch if it doesn't match any of the branches.

If the the Skip option is set to true the parser will take the branch if a match is found but will not consume the token.

If ParseOptions is omitted when creating {}ExpectToken, all options will be set to false.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

The Parser type holds a reference to the user parse func, the Scanner, AST, Errors to return and other state variables.

func (*Parser) AddNode

func (p *Parser) AddNode(nt string)

func (*Parser) AddTokens

func (p *Parser) AddTokens()

func (*Parser) Call

func (p *Parser) Call(fn func(*Parser))

func (*Parser) Exit

func (p *Parser) Exit() (AST, []Error)

func (*Parser) Expect

func (p *Parser) Expect(expect ExpectToken)

func (*Parser) GetToken

func (p *Parser) GetToken() Token

func (*Parser) Peek

func (p *Parser) Peek(branches []PeekToken)

func (*Parser) Recover

func (p *Parser) Recover(Fn func(*Parser))

func (*Parser) SkipToken

func (p *Parser) SkipToken()

func (*Parser) WalkUp

func (p *Parser) WalkUp()

type PeekToken

type PeekToken struct {
	TokenIDs []string
	Fn       func(*Parser)
}

type ScanFunc

type ScanFunc func(*Scanner) Token

type ScanOptions

type ScanOptions struct {
	Optional bool
	Multiple bool
	Invert   bool
	Error    func(*Scanner)
}

If the Optional option is false and a match is not found, an error is returned to the parser.

If the Multiple option is set to true the scanner continues to read, consume runes and take branches until a rune is read that is not matched by any of the branches or branch ranges. If the Multiple option is set to false, only the first branch (or branch range) to be matched is taken and consumed.

If the Invert option is set to true the scanner consumes the rune and takes a branch if it doesn't match any of the branch or branch ranges.

If the the Skip option is set to true the scanner will take the branch if a match is found but will not consume the rune.

If ScanOptions is omitted when creating ExpectRune{}, all options will be set to false.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

The Scanner contains a reference to the user scan function, the user input buffer, various state variables and the parser log.

func (*Scanner) Call

func (s *Scanner) Call(fn func(*Scanner))

func (*Scanner) Exit

func (s *Scanner) Exit() Token

The user scan function should return the result of Exit()

func (*Scanner) Expect

func (s *Scanner) Expect(expect ExpectRune)

Expect first reads a rune from s.read() and then tries to match it against input branches. If a match is found, the rune is 'consumed' (i.e. rune is put on the scanned buffer) and the branch is 'taken' (i.e. the branch function is called). If a match is not found, the read rune is then compared to each of the branch ranges.

If a match is not found in either the branches or branch ranges and the Optional option is set to true, the scanner returns to the user scan function without calling any of the branch functions, but still consumes the rune.

Any runes that are read but not consumed or skipped will be unread.

func (*Scanner) Match

func (s *Scanner) Match(matches []Match)

Match is required to be called by the user scan function before it returns to the user parse function, otherwise it will return the token NOT_MATCHED. Match will match every rune currently accepted by Expect() and not skipped (s.scanStr), against the input string.

Once matched a Token is generated from the input ID, s.scanStr and the current line and position of the scanner. Once the user scan function has matched a token, any subsequent calls to Match will do nothing until the user scan function returns and is called again and reset (by s.init()) by the parser.

func (*Scanner) SkipRune

func (s *Scanner) SkipRune()

type Token

type Token struct {
	ID       string
	Literal  string
	Line     int
	Position int
}

Line is the line of the source text the Token was found. Position is the position (or column) the Token was found. This information is used when displaying errors but could also be useful to the user for things like syntax highlighting and debugging if they were to implement it.

type TokenSet

type TokenSet map[string]int

func NewTokenSet

func NewTokenSet(userIds ...string) TokenSet