bonsai

README

bonsai

Pure-Go tree-sitter parsers. Each grammar is compiled to WebAssembly with a pinned toolchain and translated to Go by wasm2go, then published as its own Go module: no cgo, no shared libraries, nothing fetched at runtime.

Use

go get github.com/msuozzo/bonsai/bonsai-python

import bonsaipython "github.com/msuozzo/bonsai/bonsai-python"

src, _ := os.ReadFile("example.py")

p := bonsaipython.NewParser()
root, err := p.Parse(src)
if err != nil {
	return err
}
for fn := range root.Find("function_definition") {
	name := fn.ChildByField("name")
	fmt.Printf("def %s @ line %d\n", name.Text(src), fn.StartPoint.Row+1)
}

Import only the languages you need: every cost (download, compile, binary size) scales with the modules you actually import. A Parser is not goroutine-safe. Pool one per goroutine and reuse it across files. Instantiation is the expensive part.

Languages

module	grammar	pinned	module zip	binary cost¹
bonsai-bash	tree-sitter-bash	v0.25.1	390 KB	+2.5 MB
bonsai-dockerfile	tree-sitter-dockerfile	v0.2.0	100 KB	+0.3 MB
bonsai-go	tree-sitter-go	v0.25.0	130 KB	+0.4 MB
bonsai-gotemplate	tree-sitter-go-template	master	100 KB	+0.3 MB
bonsai-groovy	tree-sitter-groovy	initial	250 KB	+1.6 MB
bonsai-markdown	tree-sitter-markdown (block + inline)	v0.5.3	430 KB	+2.2 MB
bonsai-python	tree-sitter-python	v0.25.0	160 KB	+0.6 MB
bonsai-terraform	tree-sitter-hcl (terraform dialect)	v1.2.0	110 KB	+0.3 MB
bonsai-yaml	tree-sitter-yaml	v0.7.2	130 KB	+0.5 MB

¹ stripped-binary delta (-trimpath -ldflags='-s -w') over a 1.6 mb baseline that imports no grammar module. the bonsai-markdown row covers the combined block + inline parsers via NewFullParser, each individual grammar accounting for roughly half.

Versioning

Language modules encode the upstream grammar version in the tag's minor number and reserve the patch for re-releases (toolchain or runtime bumps at the same grammar):

bonsai-<lang>/v0.<encoded>.<respin>
encoded = major·10000 + minor·100 + patch  (of the upstream grammar tag)

tree-sitter-python v0.25.0          → bonsai-python/v0.2500.0
  …rebuilt with a newer toolchain   → bonsai-python/v0.2500.1
tree-sitter-python v0.25.1          → bonsai-python/v0.2501.0

(Mirroring upstream tags verbatim was rejected: a re-release at the same grammar version would have to squat on upstream's next patch number.) Grammars without a semver upstream tag use encoded = 0 until upstream cuts a release. The encoding exists for ordering, not parsing: the exact upstream pin is always in the module's build.env and meta_gen.go header.

The root module (github.com/msuozzo/bonsai, the runtime) versions independently as plain SemVer. Each language module pins the root version it was generated against. A runtime-ABI change means regenerating and re-releasing every language (CI enforces the regeneration half).

Releases are cut by tagging the root, replacing each language go.mod's replace directive with a require of that tag, then tagging each language module. No versions are tagged yet.

Regenerating

Each language module's *_gen.* files are produced hermetically in Docker from the pins in bonsai-<lang>/build.env:

go generate ./bonsai-python      # or: build/regen.sh python | all

CI rebuilds every language in a parallel job matrix on each PR and fails if any generated byte differs from what's checked in.

Note: Regeneration currently builds wasm2go from a fork (msuozzo/wasm2go, pinned by commit in Dockerfile.builder) that carries a fix not yet upstreamed (the approach in ncruces/wasm2go#40). It reverts to a tagged ncruces/wasm2go release once the fix lands upstream.

Licensing

The wasm→Go translation is mechanical: upstream authors retain copyright over the code compiled into each module. Each generated module is a derivative of the tree-sitter runtime and of its grammar (both MIT), so their license texts ship inside the module (LICENSE.tree-sitter, LICENSE.grammar) and are additionally embedded into consuming binaries via licenses_gen.go (LicenseTreeSitter/LicenseGrammar), keeping the required notices attached to compiled redistributions.

Documentation

Index

• Variables
• type HostEnv
  • func (*HostEnv) Xfputc(int32, int32) int32
  • func (h *HostEnv) Xmemory() MemoryAPI
  • func (*HostEnv) Xts_language_is_wasm(int32) int32
  • func (*HostEnv) Xts_wasm_store_delete(int32)
  • func (*HostEnv) Xvfprintf(int32, int32, int32) int32
  • func (*HostEnv) Xvsnprintf(int32, int32, int32, int32) int32
• type Memory
  • func (m *Memory) Grow(delta, _ int64) int64
  • func (m *Memory) Slice() *[]byte
• type MemoryAPI
• type Module
• type Node
  • func (n *Node) ChildByField(name string) *Node
  • func (n *Node) Find(typ string) iter.Seq[*Node]
  • func (n *Node) HasError() bool
  • func (n *Node) Text(src []byte) []byte
  • func (n *Node) Walk() iter.Seq[*Node]
• type Parser
  • func NewFromModule(mod Module, mem *Memory, langPtr int32) *Parser
  • func (p *Parser) Parse(src []byte) (*Node, error)
  • func (p *Parser) With(subs ...SubParser) *Parser
• type Point
• type SubParser

Variables

var ErrParseFailed = errors.New("bonsai: parse failed")

ErrParseFailed is returned by Parse when tree-sitter rejects the input outright (returns a NULL tree pointer). The grammar is permissive enough that this is rare. Most malformed source yields a tree containing error nodes rather than a failed parse.

Types

type HostEnv

type HostEnv struct {
	Mem *Memory
}

HostEnv satisfies the Xenv interface that every tree-sitter wasm module imports. Across grammars these imports are identical (memory plus a few stubs we never reach at runtime) so one HostEnv works for every grammar without per-grammar wrappers.

func (*HostEnv) Xfputc

func (*HostEnv) Xfputc(int32, int32) int32

Stdio stubs. tree-sitter's parser.c references these from debug-only branches we never trigger (no dot_graph_file is set from our exported API). Returning 0 satisfies the import.

func (*HostEnv) Xmemory

func (h *HostEnv) Xmemory() MemoryAPI

Xmemory returns the host-backed linear memory to the wasm module.

func (*HostEnv) Xts_language_is_wasm

func (*HostEnv) Xts_language_is_wasm(int32) int32

Xts_language_is_wasm: every grammar is statically linked into its wasm module, not loaded from a runtime .wasm via wasm_store. Always false.

func (*HostEnv) Xts_wasm_store_delete

func (*HostEnv) Xts_wasm_store_delete(int32)

Xts_wasm_store_delete: unreachable, never called when is_wasm is false.

func (*HostEnv) Xvfprintf

func (*HostEnv) Xvfprintf(int32, int32, int32) int32

func (*HostEnv) Xvsnprintf

func (*HostEnv) Xvsnprintf(int32, int32, int32, int32) int32

type Memory

type Memory struct {
	Buf []byte
	Max int64 // ceiling in 64 KiB pages. Grow returns -1 past this.
}

Memory backs the WASM module's linear memory.

The portable implementation grows by appending Go-side, which can relocate the backing array. Every helper in marshal.go re-fetches the slice through Slice() before each access. Do not cache the byte slice across allocations.

func (*Memory) Grow

func (m *Memory) Grow(delta, _ int64) int64

func (*Memory) Slice

func (m *Memory) Slice() *[]byte

type MemoryAPI

type MemoryAPI = interface {
	Slice() *[]byte
	Grow(delta, max int64) int64
}

MemoryAPI is the interface the wasm-module side calls when it needs linear memory. Declared as a type alias (note the `=`) so that, when HostEnv.Xmemory returns MemoryAPI, the return type unifies with each grammar's generated `Memory` alias and HostEnv structurally satisfies every grammar's Xenv interface.

type Module

type Module interface {
	X_initialize()

	Xmalloc(int32) int32
	Xfree(int32)

	Xts_parser_new() int32
	Xts_parser_delete(int32)
	Xts_parser_set_language(parser, language int32) int32
	Xts_parser_parse_string(parser, oldTree, src, length int32) int32

	Xts_tree_delete(int32)
	Xts_tree_root_node(sret, tree int32)

	Xts_node_type(sret int32) int32
	Xts_node_start_byte(sret int32) int32
	Xts_node_end_byte(sret int32) int32
	Xts_node_is_named(sret int32) int32
	Xts_node_is_error(sret int32) int32
	Xts_node_is_missing(sret int32) int32
	Xts_node_has_error(sret int32) int32
	Xts_node_start_point(sret, node int32)
	Xts_node_end_point(sret, node int32)

	Xts_tree_cursor_new(sret, node int32)
	Xts_tree_cursor_delete(cursor int32)
	Xts_tree_cursor_current_node(sret, cursor int32)
	Xts_tree_cursor_current_field_name(cursor int32) int32
	Xts_tree_cursor_goto_first_child(cursor int32) int32
	Xts_tree_cursor_goto_next_sibling(cursor int32) int32
	Xts_tree_cursor_goto_parent(cursor int32) int32
}

Module is the subset of every tree-sitter wasm module that the bonsai runtime invokes during parsing. Each grammar's generated *Module structurally implements it. There is no interface declaration on the generated side. Adding a method here requires every grammar's wasm to export the matching tree-sitter symbol.

The grammar-specific language pointer (e.g. Xtree_sitter_python) is NOT part of this interface. Each grammar's NewParser fetches it from its own *Module and passes it to NewFromModule as a plain int32.

type Node

type Node struct {
	Type      string
	Named     bool
	Field     string // field name in parent ("" if unnamed slot)
	IsError   bool   // this node is the ERROR sentinel
	IsMissing bool   // this node was inserted to recover from a parse error

	StartByte  uint32
	EndByte    uint32
	StartPoint Point
	EndPoint   Point
	Parent     *Node // nil at root
	Children   []*Node
	// contains filtered or unexported fields
}

Node is a fully-owned snapshot of a tree-sitter node. There is no live dependency on WASM memory: after Parse returns, the tree has been deleted and the Go GC owns every Node.

func (*Node) ChildByField

func (n *Node) ChildByField(name string) *Node

ChildByField returns the first child whose Field name equals name, or nil.

func (*Node) Find

func (n *Node) Find(typ string) iter.Seq[*Node]

Find yields, in preorder, every descendant of n whose Type == typ. n itself is included if it matches.

func (*Node) HasError

func (n *Node) HasError() bool

HasError reports whether the subtree rooted at n contains any error or missing nodes. Equivalent to tree-sitter's ts_node_has_error.

func (*Node) Text

func (n *Node) Text(src []byte) []byte

Text returns this node's slice of the original source. The snapshot only stores byte offsets, and the caller keeps the source buffer.

func (*Node) Walk

func (n *Node) Walk() iter.Seq[*Node]

Walk yields every descendant of n in preorder, including n.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser owns one wasm module instance plus a TSParser*. NOT safe for concurrent use. Pool a Parser per goroutine. Reuse a Parser across files. Module instantiation is the expensive part, and once warm a Parser holds onto its linear-memory high-water mark.

Grammar-specific construction lives in each grammar's package (e.g. bonsai-python.NewParser).

func NewFromModule

func NewFromModule(mod Module, mem *Memory, langPtr int32) *Parser

NewFromModule wires up a parser around an already-instantiated wasm module and the language pointer the grammar exposes. Called by each grammar package after constructing its module.

func (*Parser) Parse

func (p *Parser) Parse(src []byte) (*Node, error)

Parse builds an owned Node snapshot from src. After Parse returns, no wasm-side data backs the snapshot. The tree has been freed.

If p was configured with sub-parsers via With, Parse applies them automatically, returning the unified tree.

Parse is not concurrent-safe with itself on the same Parser.

func (*Parser) With

func (p *Parser) With(subs ...SubParser) *Parser

With registers sub-parsers on p. After With returns, p.Parse(src) applies the rules automatically and returns one unified tree. Returns p for chaining.

Markdown is the canonical case (block grammar dispatching to inline grammar):

full := bonsaimarkdown.NewParser().With(bonsai.SubParser{
	Match:  func(n *bonsai.Node) bool { return n.Type == "inline" },
	Parser: bonsaimarkdowninline.NewParser(),
})
root, _ := full.Parse(src) // tree contains block + inline nodes

Subtrees are themselves walked for further matches, so multi-layer parsing (e.g. HTML containing JS containing a template literal containing SQL) composes naturally. Beware a Match that fires on its own grammar's output: it would recurse forever. (Specifically, recursion only re-checks the CHILDREN of a replaced node, so a sub-parser whose root has the same Type as the original match is safe.)

type Point

type Point struct {
	Row uint32
	Col uint32
}

Point is a (row, column) source position. Columns are in bytes, not runes, matching the convention tree-sitter uses.

type SubParser

type SubParser struct {
	Match  func(*Node) bool
	Parser *Parser
}

SubParser pairs a node predicate with the parser that should re- parse the matched node's text. The first SubParser whose Match returns true wins for any given node.