bonsai

README

bonsai

Pure-Go tree-sitter parsers. Each grammar is compiled to WebAssembly with a pinned toolchain and translated to Go by wasm2go, then published as its own Go module: no cgo, no shared libraries, nothing fetched at runtime.

Use

go get github.com/msuozzo/bonsai/bonsai-python

import bonsaipython "github.com/msuozzo/bonsai/bonsai-python"

src, _ := os.ReadFile("example.py")

p := bonsaipython.NewParser()
root, err := p.Parse(src)
if err != nil {
	return err
}
for fn := range root.Find("function_definition") {
	name := fn.ChildByField("name")
	fmt.Printf("def %s @ line %d\n", name.Text(src), fn.StartPoint.Row+1)
}

Import only the languages you need: every cost (download, compile, binary size) scales with the modules you actually import. A Parser is not goroutine-safe. Pool one per goroutine and reuse it across files. Instantiation is the expensive part.

Languages

module	grammar	pinned	module zip	binary cost¹
bonsai-bash	tree-sitter-bash	v0.25.1	360 KB	+3.0 MB
bonsai-c	tree-sitter-c	v0.24.2	180 KB	+1.6 MB
bonsai-dockerfile	tree-sitter-dockerfile	v0.2.0	100 KB	+0.7 MB
bonsai-go	tree-sitter-go	v0.25.0	120 KB	+0.8 MB
bonsai-gotemplate	tree-sitter-go-template	master	100 KB	+0.7 MB
bonsai-groovy	tree-sitter-groovy	initial	210 KB	+1.9 MB
bonsai-java	tree-sitter-java	v0.23.5	130 KB	+1.0 MB
bonsai-javascript	tree-sitter-javascript	v0.25.0	140 KB	+1.1 MB
bonsai-kotlin	tree-sitter-kotlin	v1.1.0	460 KB	+5.0 MB
bonsai-markdown	tree-sitter-markdown (block + inline)	v0.5.3	380 KB	+3.1 MB
bonsai-python	tree-sitter-python	v0.25.0	150 KB	+1.1 MB
bonsai-ruby	tree-sitter-ruby	v0.23.1	290 KB	+3.1 MB
bonsai-rust	tree-sitter-rust	v0.24.2	200 KB	+1.8 MB
bonsai-terraform	tree-sitter-hcl (terraform dialect)	v1.2.0	110 KB	+0.7 MB
bonsai-tsx	tree-sitter-typescript (tsx dialect)	v0.23.2	230 KB	+2.1 MB
bonsai-typescript	tree-sitter-typescript (typescript dialect)	v0.23.2	220 KB	+2.1 MB
bonsai-yaml	tree-sitter-yaml	v0.7.2	130 KB	+0.9 MB

¹ stripped-binary delta (-trimpath -ldflags='-s -w') over a 1.7 mb baseline that imports no grammar module. the bonsai-markdown row covers the combined block + inline parsers via NewFullParser, each individual grammar accounting for roughly half.

Versioning

All modules, the runtime root and every language alike, release in lockstep: one vX.Y.Z tag per module per release, with every cross-module require pinned to exactly that version. The version number carries no grammar information. Each language module records its upstream pin in build.env and as generated constants (GrammarVersion, TreeSitterVersion).

Release policy and mechanics: RELEASING.md.

Regenerating

Each language module's *_gen.* files are produced hermetically in Docker from the pins in bonsai-<lang>/build.env:

go generate ./bonsai-python      # or: build/regen.sh python | all

CI rebuilds every language in a parallel job matrix on each PR and fails if any generated byte differs from what's checked in.

Licensing

The wasm→Go translation is mechanical: upstream authors retain copyright over the code compiled into each module. Each generated module is a derivative of the tree-sitter runtime and of its grammar (both MIT), so their license texts ship inside the module (LICENSE.tree-sitter, LICENSE.grammar) and are additionally embedded into consuming binaries via licenses_gen.go (LicenseTreeSitter/LicenseGrammar), keeping the required notices attached to compiled redistributions.

Documentation

Index

• Variables
• type HostEnv
  • func (*HostEnv) Xfputc(int32, int32) int32
  • func (h *HostEnv) Xmemory() MemoryAPI
  • func (*HostEnv) Xts_language_is_wasm(int32) int32
  • func (*HostEnv) Xts_wasm_store_delete(int32)
  • func (*HostEnv) Xvfprintf(int32, int32, int32) int32
  • func (*HostEnv) Xvsnprintf(int32, int32, int32, int32) int32
• type Memory
  • func (m *Memory) Grow(delta, _ int64) int64
  • func (m *Memory) Slice() *[]byte
• type MemoryAPI
• type Module
• type Node
  • func (n *Node) ChildByField(name string) *Node
  • func (n *Node) Find(typ string) iter.Seq[*Node]
  • func (n *Node) HasError() bool
  • func (n *Node) Text(src []byte) []byte
  • func (n *Node) Walk() iter.Seq[*Node]
• type Parser
  • func NewFromModule(mod Module, mem *Memory, langPtr int32) *Parser
  • func (p *Parser) Parse(src []byte) (*Node, error)
  • func (p *Parser) With(subs ...SubParser) *Parser
• type Point
• type SubParser

Variables

var ErrParseFailed = errors.New("bonsai: parse failed")

ErrParseFailed is returned by Parse when tree-sitter rejects the input outright (returns a NULL tree pointer). The grammar is permissive enough that this is rare. Most malformed source yields a tree containing error nodes rather than a failed parse.

Types

type HostEnv

type HostEnv struct {
	Mem *Memory
}

HostEnv satisfies the Xenv interface that every tree-sitter wasm module imports. Across grammars these imports are identical (memory plus a few stubs we never reach at runtime) so one HostEnv works for every grammar without per-grammar wrappers.

func (*HostEnv) Xfputc

func (*HostEnv) Xfputc(int32, int32) int32

Stdio stubs. tree-sitter's parser.c references these from debug-only branches we never trigger (no dot_graph_file is set from our exported API). Returning 0 satisfies the import.

func (*HostEnv) Xmemory

func (h *HostEnv) Xmemory() MemoryAPI

Xmemory returns the host-backed linear memory to the wasm module.

func (*HostEnv) Xts_language_is_wasm

func (*HostEnv) Xts_language_is_wasm(int32) int32

Xts_language_is_wasm: every grammar is statically linked into its wasm module, not loaded from a runtime .wasm via wasm_store. Always false.

func (*HostEnv) Xts_wasm_store_delete

func (*HostEnv) Xts_wasm_store_delete(int32)

Xts_wasm_store_delete: unreachable, never called when is_wasm is false.

func (*HostEnv) Xvfprintf

func (*HostEnv) Xvfprintf(int32, int32, int32) int32

func (*HostEnv) Xvsnprintf

func (*HostEnv) Xvsnprintf(int32, int32, int32, int32) int32

type Memory

type Memory struct {
	Buf []byte
	Max int64 // ceiling in 64 KiB pages. Grow returns -1 past this.
}

Memory backs the WASM module's linear memory.

The portable implementation grows by appending Go-side, which can relocate the backing array. Every helper in marshal.go re-fetches the slice through Slice() before each access. Do not cache the byte slice across allocations.

func (*Memory) Grow

func (m *Memory) Grow(delta, _ int64) int64

func (*Memory) Slice

func (m *Memory) Slice() *[]byte

type MemoryAPI

type MemoryAPI = interface {
	Slice() *[]byte
	Grow(delta, max int64) int64
}

MemoryAPI is the interface the wasm-module side calls when it needs linear memory. Declared as a type alias (note the `=`) so that, when HostEnv.Xmemory returns MemoryAPI, the return type unifies with each grammar's generated `Memory` alias and HostEnv structurally satisfies every grammar's Xenv interface.

type Module

type Module interface {
	X_initialize()

	Xmalloc(int32) int32
	Xfree(int32)

	Xts_parser_new() int32
	Xts_parser_delete(int32)
	Xts_parser_set_language(parser, language int32) int32
	Xts_parser_parse_string(parser, oldTree, src, length int32) int32

	Xts_tree_delete(int32)
	Xts_tree_root_node(sret, tree int32)

	Xts_node_type(sret int32) int32
	Xts_node_start_byte(sret int32) int32
	Xts_node_end_byte(sret int32) int32
	Xts_node_is_named(sret int32) int32
	Xts_node_is_error(sret int32) int32
	Xts_node_is_missing(sret int32) int32
	Xts_node_has_error(sret int32) int32
	Xts_node_start_point(sret, node int32)
	Xts_node_end_point(sret, node int32)

	Xts_tree_cursor_new(sret, node int32)
	Xts_tree_cursor_delete(cursor int32)
	Xts_tree_cursor_current_node(sret, cursor int32)
	Xts_tree_cursor_current_field_name(cursor int32) int32
	Xts_tree_cursor_goto_first_child(cursor int32) int32
	Xts_tree_cursor_goto_next_sibling(cursor int32) int32
	Xts_tree_cursor_goto_parent(cursor int32) int32
}

Module is the subset of every tree-sitter wasm module that the bonsai runtime invokes during parsing. Each grammar's generated *Module structurally implements it. There is no interface declaration on the generated side. Adding a method here requires every grammar's wasm to export the matching tree-sitter symbol.

The grammar-specific language pointer (e.g. Xtree_sitter_python) is NOT part of this interface. Each grammar's NewParser fetches it from its own *Module and passes it to NewFromModule as a plain int32.

type Node

type Node struct {
	Type      string
	Named     bool
	Field     string // field name in parent ("" if unnamed slot)
	IsError   bool   // this node is the ERROR sentinel
	IsMissing bool   // this node was inserted to recover from a parse error

	StartByte  uint32
	EndByte    uint32
	StartPoint Point
	EndPoint   Point
	Parent     *Node // nil at root
	Children   []*Node
	// contains filtered or unexported fields
}

Node is a fully-owned snapshot of a tree-sitter node. There is no live dependency on WASM memory: after Parse returns, the tree has been deleted and the Go GC owns every Node.

func (*Node) ChildByField

func (n *Node) ChildByField(name string) *Node

ChildByField returns the first child whose Field name equals name, or nil.

func (*Node) Find

func (n *Node) Find(typ string) iter.Seq[*Node]

Find yields, in preorder, every descendant of n whose Type == typ. n itself is included if it matches.

func (*Node) HasError

func (n *Node) HasError() bool

HasError reports whether the subtree rooted at n contains any error or missing nodes. Equivalent to tree-sitter's ts_node_has_error.

func (*Node) Text

func (n *Node) Text(src []byte) []byte

Text returns this node's slice of the original source. The snapshot only stores byte offsets, and the caller keeps the source buffer.

func (*Node) Walk

func (n *Node) Walk() iter.Seq[*Node]

Walk yields every descendant of n in preorder, including n.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser owns one wasm module instance plus a TSParser*. NOT safe for concurrent use. Pool a Parser per goroutine. Reuse a Parser across files. Module instantiation is the expensive part, and once warm a Parser holds onto its linear-memory high-water mark.

Grammar-specific construction lives in each grammar's package (e.g. bonsai-python.NewParser).

func NewFromModule

func NewFromModule(mod Module, mem *Memory, langPtr int32) *Parser

NewFromModule wires up a parser around an already-instantiated wasm module and the language pointer the grammar exposes. Called by each grammar package after constructing its module.

func (*Parser) Parse

func (p *Parser) Parse(src []byte) (*Node, error)

Parse builds an owned Node snapshot from src. After Parse returns, no wasm-side data backs the snapshot. The tree has been freed.

If p was configured with sub-parsers via With, Parse applies them automatically, returning the unified tree.

Parse is not concurrent-safe with itself on the same Parser.

func (*Parser) With

func (p *Parser) With(subs ...SubParser) *Parser

With registers sub-parsers on p. After With returns, p.Parse(src) applies the rules automatically and returns one unified tree. Returns p for chaining.

Markdown is the canonical case (block grammar dispatching to inline grammar):

full := bonsaimarkdown.NewParser().With(bonsai.SubParser{
	Match:  func(n *bonsai.Node) bool { return n.Type == "inline" },
	Parser: bonsaimarkdowninline.NewParser(),
})
root, _ := full.Parse(src) // tree contains block + inline nodes

Subtrees are themselves walked for further matches, so multi-layer parsing (e.g. HTML containing JS containing a template literal containing SQL) composes naturally. Beware a Match that fires on its own grammar's output: it would recurse forever. (Specifically, recursion only re-checks the CHILDREN of a replaced node, so a sub-parser whose root has the same Type as the original match is safe.)

type Point

type Point struct {
	Row uint32
	Col uint32
}

Point is a (row, column) source position. Columns are in bytes, not runes, matching the convention tree-sitter uses.

type SubParser

type SubParser struct {
	Match  func(*Node) bool
	Parser *Parser
}

SubParser pairs a node predicate with the parser that should re- parse the matched node's text. The first SubParser whose Match returns true wins for any given node.