sexpr

README

sexpr (A Go S-expression parser)

This is a full rewrite and redesign of github.com/jteeuwen/gsx. It is faster and has a considerably lower memory footprint. GSX will no longer be developed and should not be used. Use this package instead. GSX remains for legacy and compatibility reasons.

This package offers a configurable S-Expression parser. It takes any input and turns it into a parse tree.

The lexer and AST builder retain filename/line/col information from the original input source. This should help with debugging potential errors during parsing or at any later stage.

Some parts can be customized before parsing, in order to alter the behaviour and output of the lexer and AST builder. For this purpose you have to supply an instance of sexpr.Syntax to the parse function. It has the following fields:

// A set of list delimiters. These are pairs of strings denoting the
// start and end of an S-expression.
// E.g.: "(", ")"
Delimiters [][2]string

// This string starts a single line comment.
// A single line comment runs until the end of a line.
// E.g: "//"
SingleLineComment string

// These strings denote what a multi-line comment starts with
// and ends with.
// E.g.: "/*", "*/"
MultiLineComment []string

// These strings determine how a string literal starts and ends.
// E.g.: "abc".
StringLit []string

// These strings determine how a raw string literal starts and ends.
// A raw string does not have its escape sequences parsed.
// E.g.: `abc`.
RawStringLit []string

// These strings determine how a char literal starts and ends.
// E.g.: 'a'.
CharLit []string

// This function should return whether or not the given
// input qualifies as a boolean.
BooleanFunc SyntaxFunc

// This function should return whether or not the given
// input qualifies as a number.
NumberFunc SyntaxFunc

A single AST tree can be used in multiple Parse() calls for different source files. Their output will then be merged with the given AST.

For an example of how to use this package, refer to sexpr_test.go.

Dependencies

None.

License

Unless otherwise stated, all of the work in this project is subject to a 1-clause BSD license. Its contents can be found in the enclosed LICENSE file.

Documentation

Index

• Constants
• func LexBoolean(l *Lexer) int
• func LexNumber(l *Lexer) (ret int)
• func Parse(ast *AST, data []byte, syntax *Syntax) (err error)
• func ParseFile(ast *AST, file string, syntax *Syntax) (err error)
• func ParseString(ast *AST, data string, syntax *Syntax) (err error)
• type AST
  • func (a *AST) String() string
• type Lexer
  • func NewLexer(data []byte, syntax *Syntax) *Lexer
  • func (l *Lexer) Accept(valid string) int
  • func (l *Lexer) AcceptIdent() int
  • func (l *Lexer) AcceptLiteral(valid string) int
  • func (l *Lexer) AcceptRun(valid string) int
  • func (l *Lexer) AcceptSpace() int
  • func (l *Lexer) AcceptUntil(valid string) int
  • func (l *Lexer) AcceptUntilLiteral(valid string) int
  • func (l *Lexer) Ignore()
  • func (l *Lexer) Next(tok *Token)
  • func (l *Lexer) NextRune() (r rune)
  • func (l *Lexer) Rewind()
  • func (l *Lexer) Skip()
• type Node
• type ParseError
  • func NewParseError(file string, line int, col uint16, f string, argv ...interface{}) *ParseError
  • func (e *ParseError) Error() string
• type Syntax
  • func (s *Syntax) IsReserved(r rune) bool
• type SyntaxFunc
• type Token
  • func (t Token) String() string
• type TokenType
  • func (tt TokenType) String() string

Constants

const EOF = -1

Functions

func LexBoolean

func LexBoolean(l *Lexer) int

TestBoolean is a builtin function which tests if the given input might qualify as a boolean. This looks for literals 'true' and 'false'.

Assign this function to Syntax.BooleanFunc if you want default behaviour.

func LexNumber

func LexNumber(l *Lexer) (ret int)

TestNumber is a builtin function which tests if the given input might qualify as a number. This is not a guarantee, but tests for a reasonable likeness.

This finds numbers of the following formats:

1234
12.34
-0.1234
+12.34
12e-12
+1E+32
0xff12AE (hexadecimal)
0b010110101 (binary)
0644 (octal)

Assign this function to Syntax.NumberFunc if you want default behaviour.

func Parse

func Parse(ast *AST, data []byte, syntax *Syntax) (err error)

Parse processes the given data and stores all the nodes it finds in the given AST instance. The parser uses the given syntax rule set to perform the parsing.

func ParseFile

func ParseFile(ast *AST, file string, syntax *Syntax) (err error)

ParseFile processes the given file and stores all the nodes it finds in the given AST instance. The parser uses the given syntax rule set to perform the parsing.

func ParseString

func ParseString(ast *AST, data string, syntax *Syntax) (err error)

ParseString processes the given data and stores all the nodes it finds in the given AST instance. The parser uses the given syntax rule set to perform the parsing.

Types

type AST

type AST struct {
	// Root node.
	Root Node

	// Name of the source files this AST was built from.
	// A single AST can be used as input for multiple parse sessions.
	// The generated data is then merged with the existing AST.
	//
	// Each node retains line/column information from the source it came from.
	// Additionally, it will have an integer index into this list of
	// file names.
	Files []string
}

An abstract syntax tree.

func (*AST) String

func (a *AST) String() string

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

A Lexer turns s-expression source into a stream of tokens.

func NewLexer

func NewLexer(data []byte, syntax *Syntax) *Lexer

New creates a new lexer for the given input data. The meaning of tokens this lexer looks for can be configured through the supplied Syntax struct.

func (*Lexer) Accept

func (l *Lexer) Accept(valid string) int

Accept consumes the next rune if it is contained in the supplied string.

func (*Lexer) AcceptIdent

func (l *Lexer) AcceptIdent() int

AcceptIdent consumes runes until it hits anything that does not qualify as a valid identifier, or is one of the reserved tokens in our syntax struct.

func (*Lexer) AcceptLiteral

func (l *Lexer) AcceptLiteral(valid string) int

AcceptLiteral consumes runes if they are an exact, rune-for-rune match with the supplied string.

func (*Lexer) AcceptRun

func (l *Lexer) AcceptRun(valid string) int

AcceptRun consumes runes for as long they are contained in the supplied string. It returns the number of runes consumed or EOF.

func (*Lexer) AcceptSpace

func (l *Lexer) AcceptSpace() int

AcceptSpace consumes runes for as long as they are whitespace.

func (*Lexer) AcceptUntil

func (l *Lexer) AcceptUntil(valid string) int

AcceptUntil consumes runes for as long they are NOT contained in the supplied string.

func (*Lexer) AcceptUntilLiteral

func (l *Lexer) AcceptUntilLiteral(valid string) int

AcceptUntilLiteral consumes runes for as long they are not an exact, rune-for-rune match with the supplied string.

func (*Lexer) Ignore

func (l *Lexer) Ignore()

Ignore the input so far.

func (*Lexer) Next

func (l *Lexer) Next(tok *Token)

Next returns the next token. If there are none available, this yields a token with Type set to TokEof. TokErr denotes that an error occurred.

func (*Lexer) NextRune

func (l *Lexer) NextRune() (r rune)

NextRune retuns the nextrune unicode rune in the input.

func (*Lexer) Rewind

func (l *Lexer) Rewind()

Rewind Rewinds to the last rune. Can be called only once per NextRune() call.

func (*Lexer) Skip

func (l *Lexer) Skip()

Skip Skips the NextRune character.

type Node

type Node struct {
	Data     []byte    // Node data.
	Children []*Node   // Optional child nodes.
	Parent   *Node     // Parent node.
	Line     int       // Line in original source file.
	Col      uint16    // Column in original source file.
	File     uint8     // Index of name for original source file.
	Type     TokenType // Type of node.
}

An AST node

type ParseError

type ParseError struct {
	Line int
	Col  uint16
	File string
	Msg  string
}

Represents a parse error.

func NewParseError

func NewParseError(file string, line int, col uint16, f string, argv ...interface{}) *ParseError

NewParseError creates a new parse error from the given values.

func (*ParseError) Error

func (e *ParseError) Error() string

Error returns a string representation of this error.

type Syntax

type Syntax struct {
	// A set of list delimiters. These are pairs of strings denoting the
	// start and end of an S-expression.
	// E.g.: "(", ")"
	Delimiters [][2]string

	// This string starts a single line comment.
	// A single line comment runs until the end of a line.
	// E.g: "//"
	SingleLineComment string

	// These strings denote what a multi-line comment starts with
	// and ends with.
	// E.g.: "/*", "*/"
	MultiLineComment []string

	// These strings determine how a string literal starts and ends.
	// E.g.: "abc".
	StringLit []string

	// These strings determine how a raw string literal starts and ends.
	// A raw string does not have its escape sequences parsed.
	// E.g.: `abc`.
	RawStringLit []string

	// These strings determine how a char literal starts and ends.
	// E.g.: 'a'.
	CharLit []string

	// This function should return whether or not the given
	// input qualifies as a boolean.
	BooleanFunc SyntaxFunc

	// This function should return whether or not the given
	// input qualifies as a number.
	NumberFunc SyntaxFunc
}

A Syntax struct contains rules on how the lexer should treat the characters it encounters in the source. This determines what tokens are generated.

func (*Syntax) IsReserved

func (s *Syntax) IsReserved(r rune) bool

IsReserved returns true if the given rune is contained in one of the syntax fields.

type SyntaxFunc

type SyntaxFunc func(*Lexer) int

type Token

type Token struct {
	Data []byte
	Line int
	Col  uint16
	Type TokenType
}

func (Token) String

func (t Token) String() string

type TokenType

type TokenType uint8

const (
	TokListOpen TokenType = iota
	TokListClose
	TokComment
	TokIdent
	TokString
	TokRawString
	TokChar
	TokNumber
	TokBoolean
	TokEof
	TokErr
)

func (TokenType) String

func (tt TokenType) String() string