Help us shape the future of Go! Take the 2026 Go Developer Survey

sexpr

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 3, 2026 License: MIT Imports: 12 Imported by: 1

README

sexpr

A library for working with S-expressions in Go.

Install

go get github.com/z5labs/sexpr-go

Quickstart

Parse some source into an AST, walk it, and print it back out:

package main

import (
	"fmt"
	"os"
	"strings"

	"github.com/z5labs/sexpr-go"
)

func main() {
	f, err := sexpr.Parse(strings.NewReader(`(add 1 2.5 "hi" #t nil)`))
	if err != nil {
		panic(err)
	}

	list := f.Nodes[0].(sexpr.List)
	for _, elem := range list.Elements {
		switch n := elem.(type) {
		case sexpr.Symbol:
			fmt.Printf("symbol %s\n", n.Value)
		case sexpr.Int:
			fmt.Printf("int %d\n", n.Value)
		case sexpr.Float:
			fmt.Printf("float %v\n", n.Value)
		case sexpr.String:
			fmt.Printf("string %q\n", n.Value)
		case sexpr.Bool:
			fmt.Printf("bool %t\n", n.Value)
		case sexpr.Nil:
			fmt.Println("nil")
		}
	}

	if err := sexpr.Print(os.Stdout, f); err != nil {
		panic(err)
	}
}

There are three entry points, layered on each other:

Function Purpose
sexpr.Tokenize(r io.Reader) iter.Seq2[sexpr.Token, error] Scan a reader into tokens, keeping each lexeme's source text and position.
sexpr.Parse(r io.Reader) (*sexpr.File, error) Build an AST, decoding string escapes and number literals.
sexpr.Print(w io.Writer, f *sexpr.File) error Write an AST back out as S-expression text.

Print followed by Parse gives back the same values, so the two round trip.

Runnable versions of all of these live in example_test.go.

Supported grammar

A source is any number of top level datums. A datum is one of:

Kind Examples Node type
symbol add, x, +, ->list sexpr.Symbol
number 42, -7, 0.5, -1.5e-3 sexpr.Int or sexpr.Float
string "hello", "a\nb" sexpr.String
boolean #t, #true, #f, #false sexpr.Bool
nil nil sexpr.Nil
list (), (add 1 2), (a b . c) sexpr.List
quoted 'x, `x, ,x, ,@x sexpr.Quote
  • Symbols are runs of letters, digits, and the punctuation +-*/<>=!?:$%_&~^@. which are neither a number nor the single character .. Letters include non-ASCII ones, so π is a symbol.
  • Numbers are an optional sign, then digits with an optional fraction and an optional decimal exponent. Digits are ASCII only. A decimal point must have a digit after it, so 1. is not a number, and an exponent needs at least one digit. A lexeme that starts like a number but is malformed — 12abc, --1 — is an error rather than a symbol. A literal with neither fraction nor exponent parses as an Int, everything else as a Float.
  • Strings are double quoted and support the escapes \" \\ \n \r \t \b \f and \uXXXX. Any other escape is an error.
  • Lists may be improper: a . before the last datum puts it in List.Tail instead of List.Elements. The empty list () is distinct from nil.
  • Reader macros apply to the datum that follows. The shorthand is kept rather than rewritten to (quote x), so printing reproduces what was written.
  • Comments come in two forms — ; to end of line, and #| ... |#, which spans lines and nests. Both are kept, attached to the File or List that encloses them.

Nesting is bounded by MaxDepth, so a pathological input fails with a MaxDepthExceededError instead of exhausting the stack.

Out of scope

This package covers the common core of S-expression syntax, not any one Lisp's full reader. None of the following is supported, and each is a syntax error rather than a silently accepted extension:

  • [...] bracket lists
  • #; datum comments
  • #\char character literals
  • #(...) vectors
  • |escaped symbols|
  • the full numeric tower — rationals, complex numbers, arbitrary precision integers, and the #x, #o, #b, #e, and #i prefixes. Numbers are decimal, and are an int64 or a float64.
  • the canonical and base64 transport encodings of the S-expression internet draft

Marshalling between S-expressions and Go structs is also out of scope. This package gives you the syntax tree; mapping it onto your own types is yours to do.

License

MIT

Documentation

Overview

Package sexpr provides tools for working with S-expressions.

Entry points

There are three, layered on each other:

  • Tokenize scans a reader into an iter.Seq2 of Token values, keeping the source text of each lexeme. Use it when you want the lexemes and their positions without an AST.
  • Parse builds a File from a reader, turning tokens into Node values and decoding string escapes and number literals along the way.
  • Print writes a File back out as S-expression text. Parsing what Print produced yields the same values it was given.

Grammar

A source is any number of top level datums. A datum is one of: 

symbol      add, x, +, ->list, hello-world
number      42, -7, 0.5, -1.5e-3
string      "hello", "a\nb", "é"
boolean     #t, #true, #f, #false
nil         nil
list        (), (add 1 2), (a b . c)
quoted      'x, `x, ,x, ,@x

A symbol is a run of letters, digits, and the punctuation "+-*/<>=!?:$%_&~^@." which is not a number and is not the single character ".". Letters include non-ASCII ones, so π is a symbol.

A number is an optional sign, then digits with an optional fraction and an optional decimal exponent. Digits are ASCII only. A decimal point must have a digit after it, so "1." is not a number, and an exponent must have at least one digit. A lexeme which begins like a number but is malformed, such as "12abc" or "--1", is an error rather than a symbol. Parse yields an Int for a literal with neither fraction nor exponent and a Float otherwise.

A string is delimited by double quotes and may contain the escapes \" \\ \n \r \t \b \f and \uXXXX, where X is a hex digit. Any other escape is an error. Tokenize keeps the escapes as written; Parse decodes them into String.Value.

A list is a parenthesized sequence of datums. Writing a "." before the last datum makes an improper list, whose tail is held in List.Tail rather than in List.Elements. The empty list "()" is distinct from Nil.

The reader macros "'", "`", ",", and ",@" each apply to the datum which follows, producing a Quote. The shorthand is kept rather than rewritten to (quote x), so printing reproduces what was written.

Comments come in two forms: ";" runs to the end of the line, and "#| ... |#" spans lines and nests. Both are kept, attached to the File or List which encloses them.

Nesting is bounded by MaxDepth, which Parse enforces so a pathological input fails with a MaxDepthExceededError rather than exhausting the stack.

Out of scope

This package covers the common core of S-expression syntax, not any one Lisp's full reader. None of the following is supported, and each is a syntax error rather than a silently accepted extension:

  • "[...]" bracket lists
  • "#;" datum comments
  • "#\char" character literals
  • "#(...)" vectors
  • "|escaped symbols|"
  • the full numeric tower: rationals, complex numbers, arbitrary precision integers, and the "#x", "#o", "#b", "#e", and "#i" prefixes. Numbers are decimal, and are an int64 or a float64.
  • the canonical and base64 transport encodings of the S-expression internet draft

Marshalling between S-expressions and Go structs is also out of scope. This package gives you the syntax tree; mapping it onto your own types is yours to do.

Printing layout

Print writes each top level datum on its own line.

A list is written on one line, with its elements separated by a single space, whenever that line would end at or before MaxLineWidth: 

(add 1 2)

When it would run past that, the list breaks onto one element per line. The head stays beside the opening parenthesis and the remaining elements are indented two spaces past it, so they line up under the head: 

(function-with-a-long-name
  argument-one
  argument-two)

The indent is measured from the list's own opening parenthesis, so nesting compounds it: a list which starts at column 2 indents its elements to 4. Only the lists which do not fit are broken, so a short inner list stays on one line inside a broken outer one.

The tail of a dotted pair follows the same rule, written on its own line after the dot: 

(aaa
  bbb
  . ccc)

Quote shorthands are reproduced as written, never expanded to (quote x), and a broken datum after one is indented past the macro.

An atom is never broken, so a string or symbol longer than MaxLineWidth simply runs past it.

Comments

Comments are written back in source order among the datums of whichever container holds them, each on its own line: 

; leading
(add 1 2)
; trailing

A line comment is always followed by a newline, so it can never swallow the code after it. A block comment is written exactly as it appeared, delimiters included.

Because of that, a list holding any comment is always broken across lines even when it would otherwise fit, and a comment written before a list's first element pushes that element off the opening line: 

(
  ; about a
  a
  b)

A comment after the last element likewise moves the closing parenthesis onto its own line.

Index

Examples

Constants

View Source 
const MaxDepth = 10_000

MaxDepth is the deepest list nesting Parse accepts.

Parsing is recursive descent, so nesting costs stack. Rather than let a pathological input exhaust it, anything deeper than this fails with a MaxDepthExceededError.

View Source 
const MaxLineWidth = 80

MaxLineWidth is the column past which Print breaks a list across lines.

Width is counted in bytes, matching how Pos counts columns.

Variables

View Source 
var ErrNilComment = errors.New("cannot print a nil comment")

ErrNilComment is returned by Print when a container holds a nil Comment.

View Source 
var ErrNilFile = errors.New("cannot print a nil file")

ErrNilFile is returned by Print when given a nil File.

Functions

func Print 

func Print(w io.Writer, f *File) error

Print the given File to the given writer as S-expression text.

Example 
package main

import (
	"fmt"
	"os"

	"github.com/z5labs/sexpr-go"
)

func main() {
	f := &sexpr.File{
		Nodes: []sexpr.Node{
			sexpr.List{
				Elements: []sexpr.Node{
					sexpr.Symbol{Value: "add"},
					sexpr.Int{Value: 1},
					sexpr.Int{Value: 2},
				},
			},
			sexpr.List{
				Elements: []sexpr.Node{
					sexpr.Symbol{Value: "define"},
					sexpr.Symbol{Value: "a-name-long-enough-to-need-breaking"},
					sexpr.String{Value: "a value long enough to push past the line width"},
				},
			},
		},
	}

	if err := sexpr.Print(os.Stdout, f); err != nil {
		fmt.Println(err)
	}

}

Output:
(add 1 2)
(define
  a-name-long-enough-to-need-breaking
  "a value long enough to push past the line width")

func Tokenize 

func Tokenize(r io.Reader) iter.Seq2[Token, error]

Tokenize the S-expression defined in the given reader.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/z5labs/sexpr-go"
)

func main() {
	src := `(add 1 2.5 "hi" #t) ; done`

	for tok, err := range sexpr.Tokenize(strings.NewReader(src)) {
		if err != nil {
			fmt.Println(err)
			return
		}
		fmt.Printf("%d:%d %s\n", tok.Pos.Line, tok.Pos.Column, tok)
	}

}

Output:
1:1 LParen(()
1:2 Symbol(add)
1:6 Number(1)
1:8 Number(2.5)
1:12 String(hi)
1:17 Bool(#t)
1:19 RParen())
1:21 Comment(; done)

Types

type Bool 

type Bool struct {
	Pos   Pos
	Value bool
}

Bool represents a boolean literal, written #t, #true, #f, or #false.

type Comment 

type Comment struct {
	Pos  Pos
	Text string
}

Comment represents a comment in the source.

type File 

type File struct {
	Nodes    []Node
	Comments []*Comment
}

File represents a parsed S-expression source, which may hold any number of top level datums.

func Parse 

func Parse(r io.Reader) (*File, error)

Parse the S-expressions defined in the given reader.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/z5labs/sexpr-go"
)

func main() {
	src := `(add 1 2.5 "hi" #t nil)`

	f, err := sexpr.Parse(strings.NewReader(src))
	if err != nil {
		fmt.Println(err)
		return
	}

	list := f.Nodes[0].(sexpr.List)
	for _, elem := range list.Elements {
		switch n := elem.(type) {
		case sexpr.Symbol:
			fmt.Printf("symbol %s\n", n.Value)
		case sexpr.Int:
			fmt.Printf("int %d\n", n.Value)
		case sexpr.Float:
			fmt.Printf("float %v\n", n.Value)
		case sexpr.String:
			fmt.Printf("string %q\n", n.Value)
		case sexpr.Bool:
			fmt.Printf("bool %t\n", n.Value)
		case sexpr.Nil:
			fmt.Println("nil")
		}
	}

}

Output:
symbol add
int 1
float 2.5
string "hi"
bool true
nil
Example (Errors) 
package main

import (
	"fmt"
	"strings"

	"github.com/z5labs/sexpr-go"
)

func main() {
	_, err := sexpr.Parse(strings.NewReader(`(add 1`))
	if err != nil {
		fmt.Println(err)
	}

	_, err = sexpr.Parse(strings.NewReader(`(add 1 . )`))
	if err != nil {
		fmt.Println(err)
	}

	_, err = sexpr.Parse(strings.NewReader(`(add 12abc)`))
	if err != nil {
		fmt.Println(err)
	}

	_, err = sexpr.Parse(strings.NewReader(`(add "oops)`))
	if err != nil {
		fmt.Println(err)
	}

}

Output:
unexpected end of tokens at line 1, column 6, expected one of: RParen
unexpected token at line 1, column 10: RParen()), expected one of: LParen, Quote, Symbol, String, Number, Bool
invalid number literal "12abc" at line 1, column 6
unterminated string literal at line 1, column 6

type Float 

type Float struct {
	Pos   Pos
	Value float64
}

Float represents a floating point literal.

type Int 

type Int struct {
	Pos   Pos
	Value int64
}

Int represents an integer literal.

type InvalidEscapeError 

type InvalidEscapeError struct {
	Pos Pos
	R   rune
}

InvalidEscapeError is the error returned by the tokenizer when a string literal contains an unrecognized escape sequence.

func (InvalidEscapeError) Error 

func (e InvalidEscapeError) Error() string

Error implements the [error] interface.

type InvalidNumberError 

type InvalidNumberError struct {
	Pos   Pos
	Value string
}

InvalidNumberError is the error returned by the tokenizer when a lexeme is meant to be a number but is malformed.

func (InvalidNumberError) Error 

func (e InvalidNumberError) Error() string

Error implements the [error] interface.

type InvalidSymbolError 

type InvalidSymbolError struct {
	// Value comes first because the message leads with it, and because being
	// field-identical to [Symbol] is a coincidence worth not relying on.
	Value string
	Pos   Pos
}

InvalidSymbolError is the error returned by the printer when a Symbol holds text which is not a symbol.

Symbols are written verbatim because the format has no way to quote them, so a value such as "123" or "a b" would read back as something other than the symbol it came from.

func (InvalidSymbolError) Error 

func (e InvalidSymbolError) Error() string

Error implements the [error] interface.

type List 

type List struct {
	Pos      Pos
	Elements []Node
	Tail     Node

	// Comments holds the comments written inside this list, in the order they
	// appear. They are attached to the list rather than to any one element.
	Comments []*Comment
}

List represents a parenthesized list. An empty list has no elements and is distinct from Nil.

Tail holds the datum after the dot of an improper list such as (a . b), and is nil for a proper list. Elements always holds what precedes the dot, so (a b . c) has two elements and a tail.

type MaxDepthExceededError 

type MaxDepthExceededError struct {
	Pos   Pos
	Depth int
}

MaxDepthExceededError is the error returned by the parser when the input nests deeper than MaxDepth.

func (MaxDepthExceededError) Error 

func (e MaxDepthExceededError) Error() string

Error implements the [error] interface.

type Nil 

type Nil struct {
	Pos Pos
}

Nil represents the empty value, written nil.

type Node 

type Node interface {
	// contains filtered or unexported methods
}

Node represents a node in the S-expression AST.

The interface is sealed by an unexported method, so only this package can implement it and a type switch over the node types is exhaustive.

type NonFiniteFloatError 

type NonFiniteFloatError struct {
	Value float64
}

NonFiniteFloatError is the error returned by the printer when a Float holds an infinity or a NaN.

S-expressions have no syntax for those, so writing one would produce output which reads back as a symbol rather than a number.

func (NonFiniteFloatError) Error 

func (e NonFiniteFloatError) Error() string

Error implements the [error] interface.

type NumberRangeError 

type NumberRangeError struct {
	Pos   Pos
	Value string
}

NumberRangeError is the error returned by the parser when a number literal is well formed but cannot be represented as an int64 or a float64.

func (NumberRangeError) Error 

func (e NumberRangeError) Error() string

Error implements the [error] interface.

type Pos 

type Pos struct {
	Line   int
	Column int
}

Pos represents the position of a token in the input.

Line is 1-based. Column is a 1-based byte offset into the line, so a multi-byte rune advances Column by its encoded width rather than by one.

type Quote 

type Quote struct {
	Pos   Pos
	Kind  QuoteKind
	Datum Node
}

Quote represents a datum written with one of the reader macro shorthands.

The shorthand is kept rather than rewritten to (quote x), so that printing reproduces what was written.

type QuoteKind 

type QuoteKind int

QuoteKind identifies which reader macro produced a Quote. 

const (
	QuoteKindQuote           QuoteKind = iota // '
	QuoteKindQuasiquote                       // `
	QuoteKindUnquote                          // ,
	QuoteKindUnquoteSplicing                  // ,@
)

The reader macro shorthands a Quote may carry.

func (QuoteKind) String 

func (k QuoteKind) String() string

String implements the fmt.Stringer interface. It panics on a QuoteKind which is not one of the constants above, since that can only come from a value this package never produced.

type String 

type String struct {
	Pos   Pos
	Value string
}

String represents a string literal. Value holds the decoded text, with the escape sequences of the source resolved.

type Symbol 

type Symbol struct {
	Pos   Pos
	Value string
}

Symbol represents a symbol such as add or ->list.

type TailWithoutElementsError 

type TailWithoutElementsError struct {
	Pos Pos
}

TailWithoutElementsError is the error returned by the printer when a List carries a tail but has no elements.

A dot separates two halves, so there is no S-expression for a tail with nothing before it; writing "(. b)" would produce text the parser rejects.

func (TailWithoutElementsError) Error 

func (e TailWithoutElementsError) Error() string

Error implements the [error] interface.

type Token 

type Token struct {
	Pos   Pos
	Type  TokenType
	Value []byte
}

Token represents a token in an S-expression.

func (Token) String 

func (t Token) String() string

String implements the fmt.Stringer interface, rendering the token as its type followed by its value in parentheses, such as Symbol(add).

type TokenType 

type TokenType int

TokenType represents the type of a token. 

const (
	TokenLParen  TokenType = iota // "("
	TokenRParen                   // ")"
	TokenSymbol                   // e.g. add, x, +, ->list
	TokenComment                  // e.g. ; comment or #| comment |#
	TokenString                   // e.g. "hello"
	TokenNumber                   // e.g. 42, -0.5, 1.5e-3
	TokenDot                      // "." separating the halves of a dotted pair
	TokenBool                     // #t, #true, #f, or #false
	TokenQuote                    // a reader macro: ', `, , or ,@
)

The token types Tokenize may yield. Punctuation gets its own type rather than being lumped together, since TokenSymbol means the Lisp atom here.

func (TokenType) String 

func (tt TokenType) String() string

String implements the fmt.Stringer interface. It panics on a TokenType which is not one of the constants above, since that can only come from a value this package never produced.

type UnexpectedCharacterError 

type UnexpectedCharacterError struct {
	Pos Pos
	R   rune
}

UnexpectedCharacterError is the error returned by the tokenizer when it encounters an unexpected character in the input.

func (UnexpectedCharacterError) Error 

func (e UnexpectedCharacterError) Error() string

Error implements the [error] interface.

type UnexpectedEndOfTokensError 

type UnexpectedEndOfTokensError struct {
	Expected []TokenType
	Pos      Pos
}

UnexpectedEndOfTokensError is the error returned by the parser when it reaches the end of the tokens unexpectedly.

func (UnexpectedEndOfTokensError) Error 

func (e UnexpectedEndOfTokensError) Error() string

Error implements the [error] interface.

type UnexpectedTokenError 

type UnexpectedTokenError struct {
	Expected []TokenType
	Actual   Token
}

UnexpectedTokenError is the error returned by the parser when it encounters an unexpected token.

func (UnexpectedTokenError) Error 

func (e UnexpectedTokenError) Error() string

Error implements the [error] interface.

type UnsupportedNodeError 

type UnsupportedNodeError struct {
	Node Node
}

UnsupportedNodeError is the error returned by the printer when it meets a node it cannot write.

func (UnsupportedNodeError) Error 

func (e UnsupportedNodeError) Error() string

Error implements the [error] interface.

type UnterminatedCommentError 

type UnterminatedCommentError struct {
	Pos Pos
}

UnterminatedCommentError is the error returned by the tokenizer when a block comment is never closed.

func (UnterminatedCommentError) Error 

func (e UnterminatedCommentError) Error() string

Error implements the [error] interface.

type UnterminatedStringError 

type UnterminatedStringError struct {
	Pos Pos
}

UnterminatedStringError is the error returned by the tokenizer when a string literal is never closed.

func (UnterminatedStringError) Error 

func (e UnterminatedStringError) Error() string

Error implements the [error] interface.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.