Help us shape the future of Go! Take the 2026 Go Developer Survey

gositter

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 17, 2026 License: MIT Imports: 12 Imported by: 0

README

gositter

A Go port of the tree-sitter parser runtime, version 0.25.0. No cgo.

The goal is the same syntax tree as the C runtime for every input, broken files included. Parser tables come from each grammar's generated parser.c. External scanners are ported by hand.

Grammars: bash, c, cpp, c-sharp, go, hcl, java, javascript, json, julia, kotlin, lua, php, python, ruby, rust, scala, typescript, tsx, verilog, zig.

oracle/ is a separate module that uses cgo to compare every tree with the C runtime. The root module needs only the standard library.

Install

go get github.com/dobbo-ca/gositter

Needs Go 1.25 or later.

Usage

package main

import (
	"fmt"
	"strings"

	"github.com/dobbo-ca/gositter"
	"github.com/dobbo-ca/gositter/grammars/python"
)

func main() {
	src := []byte("def add(a, b):\n    return a + b\n")

	p := gositter.NewParser()
	if err := p.SetLanguage(python.Language()); err != nil {
		panic(err)
	}
	tree := p.Parse(src)

	var walk func(n gositter.Node, depth int)
	walk = func(n gositter.Node, depth int) {
		fmt.Printf("%s%s [%d, %d)\n", strings.Repeat("  ", depth), n.Kind(), n.StartByte(), n.EndByte())
		for _, c := range n.NamedChildren(nil) {
			walk(c, depth+1)
		}
	}
	walk(tree.RootNode(), 0)
}

The same code runs as Example in example_test.go. A Parser is not safe for concurrent use, and a tree is valid until the next Parse, Reset or SetLanguage on its parser.

ParseBudget(src, maxStall) is Parse that returns nil once the parser has advanced more than maxStall times in a row without passing the furthest byte it has reached, which bounds the time spent on pathological input.

Oracle

The oracle needs cgo and a C compiler. It parses every file with both runtimes and reports the trees that differ:

cd oracle
./build.sh
./oracle cmp python /path/to/python/files

Status

v0.x: the API can change in any minor release. Used by graphify-go.

License

MIT, see LICENSE. The runtime is a port of tree-sitter and each package under grammars/ derives from its upstream grammar. NOTICE.md lists them and the license text of each.

Documentation

Overview

Package gositter is a Go port of the tree-sitter parser runtime v0.25.0 (ABI 15). This file is the frozen contract between the runtime, the grammar generator and the hand-written scanners. See CONTRACT.md. Changes go through scratch/CONTRACT_CHANGES.md.

Example 
package main

import (
	"fmt"
	"strings"

	"github.com/dobbo-ca/gositter"
	"github.com/dobbo-ca/gositter/grammars/python"
)

func main() {
	src := []byte("def add(a, b):\n    return a + b\n")

	p := gositter.NewParser()
	if err := p.SetLanguage(python.Language()); err != nil {
		panic(err)
	}
	tree := p.Parse(src)

	var walk func(n gositter.Node, depth int)
	walk = func(n gositter.Node, depth int) {
		fmt.Printf("%s%s [%d, %d)\n", strings.Repeat("  ", depth), n.Kind(), n.StartByte(), n.EndByte())
		for _, c := range n.NamedChildren(nil) {
			walk(c, depth+1)
		}
	}
	walk(tree.RootNode(), 0)
}

Output:
module [0, 32)
  function_definition [0, 31)
    identifier [4, 7)
    parameters [7, 13)
      identifier [8, 9)
      identifier [11, 12)
    block [19, 31)
      return_statement [19, 31)
        binary_operator [26, 31)
          identifier [26, 27)
          identifier [30, 31)

Index

Examples

Constants

View Source 
const (
	SymbolEnd         Symbol = 0      // ts_builtin_sym_end
	SymbolError       Symbol = 0xFFFF // ts_builtin_sym_error
	SymbolErrorRepeat Symbol = 0xFFFE // ts_builtin_sym_error_repeat

	// TREE_SITTER_SERIALIZATION_BUFFER_SIZE
	SerializationBufferSize = 1024

	LanguageVersion                  = 15 // TREE_SITTER_LANGUAGE_VERSION
	MinCompatibleLanguageVersion     = 13 // TREE_SITTER_MIN_COMPATIBLE_LANGUAGE_VERSION
	LanguageVersionWithReservedWords = 15
	LanguageVersionWithPrimaryStates = 14
)
View Source 
const (
	ParseActionTypeShift   uint8 = 0
	ParseActionTypeReduce  uint8 = 1
	ParseActionTypeAccept  uint8 = 2
	ParseActionTypeRecover uint8 = 3
)

TSParseActionType.

Variables

This section is empty.

Functions

func SetContains 

func SetContains(ranges []CharacterRange, lookahead int32) bool

set_contains

func SetContainsASCII 

func SetContainsASCII(ascii *[128]bool, ranges []CharacterRange, lookahead int32) bool

SetContainsASCII is set_contains with the answers for code points below 128 precomputed by the generator: ascii[c] is SetContains(ranges, c).

Types

type CharacterRange 

type CharacterRange struct {
	Start int32
	End   int32
}

TSCharacterRange, inclusive on both ends.

type ExternalScanner 

type ExternalScanner interface {
	// scan. validSymbols has ExternalTokenCount elements and is read-only.
	// On success the scanner sets l.ResultSymbol to the external token index.
	Scan(l *Lexer, validSymbols []bool) bool
	// serialize. len(buf) == SerializationBufferSize. Returns bytes written.
	Serialize(buf []byte) int
	// deserialize. len(buf) == 0 means reset to the initial state. buf is
	// only valid during the call.
	Deserialize(buf []byte)
}

ExternalScanner is one instance of a grammar's external scanner. The runtime creates it where C calls create and drops it where C calls destroy.

type ExternalScannerInfo 

type ExternalScannerInfo struct {
	States    []bool   // [external state][ExternalTokenCount], row 0 unused
	SymbolMap []Symbol // external token index -> symbol
	New       func() ExternalScanner
}

TSLanguage.external_scanner. destroy has no Go counterpart.

type FieldID 

type FieldID uint16

TSSymbol, TSStateId, TSFieldId.

type FieldMapEntry 

type FieldMapEntry struct {
	FieldID    FieldID
	ChildIndex uint8
	Inherited  bool
}

TSFieldMapEntry (4 bytes).

type Language 

type Language struct {
	ABIVersion             uint32
	SymbolCount            uint32
	AliasCount             uint32
	TokenCount             uint32
	ExternalTokenCount     uint32
	StateCount             uint32
	LargeStateCount        uint32
	ProductionIDCount      uint32
	FieldCount             uint32
	MaxAliasSequenceLength uint16

	ParseTable         []uint16 // [LargeStateCount][SymbolCount]
	SmallParseTable    []uint16
	SmallParseTableMap []uint32
	ParseActions       []ParseAction

	SymbolNames     []string // SymbolCount + AliasCount entries
	FieldNames      []string // FieldCount + 1 entries, [0] == ""
	FieldMapSlices  []MapSlice
	FieldMapEntries []FieldMapEntry
	SymbolMetadata  []SymbolMetadata
	PublicSymbolMap []Symbol
	AliasMap        []uint16 // zero-terminated, as in C
	AliasSequences  []Symbol // [ProductionIDCount][MaxAliasSequenceLength]

	LexModes            []LexMode
	LexFn               LexFunc
	KeywordLexFn        LexFunc // nil if the grammar has no word token
	KeywordCaptureToken Symbol

	ExternalScanner ExternalScannerInfo

	PrimaryStateIDs []StateID
	Name            string

	ReservedWords          []Symbol // [set][MaxReservedWordSetSize]
	MaxReservedWordSetSize uint16

	SupertypeCount      uint32
	SupertypeSymbols    []Symbol
	SupertypeMapSlices  []MapSlice
	SupertypeMapEntries []Symbol

	Metadata LanguageMetadata
	// contains filtered or unexported fields
}

Language mirrors TSLanguage (ABI 15) field for field. C pointers become slices; two-dimensional C arrays are flattened row-major. A Language is shared by all parsers and, KindMask before the first parse aside, immutable after construction.

func (*Language) FieldIDForName 

func (l *Language) FieldIDForName(name string) FieldID

FieldIDForName is ts_language_field_id_for_name. The scan's answer for each field name is kept; any other name is scanned.

func (*Language) FieldNameForID 

func (l *Language) FieldNameForID(id FieldID) string

ts_language_field_name_for_id. Returns "" for 0 or out of range.

func (*Language) KindMask 

func (l *Language) KindMask(kinds ...string) uint32

KindMask returns the bits that stand for the named node kinds in Node.PreorderMask, giving each new name the next of 32 bits. A tree knows the kinds named before it was parsed, and the language is not locked: name every kind before the first Parse. No C counterpart.

func (*Language) SymbolName 

func (l *Language) SymbolName(s Symbol) string

ts_language_symbol_name

type LanguageMetadata 

type LanguageMetadata struct {
	MajorVersion uint8
	MinorVersion uint8
	PatchVersion uint8
}

TSLanguageMetadata.

type LexFunc 

type LexFunc func(l *Lexer, state StateID) bool

LexFunc is the type of a generated ts_lex / ts_lex_keywords.

type LexMode 

type LexMode struct {
	LexState          uint16
	ExternalLexState  uint16
	ReservedWordSetID uint16
}

TSLexerMode. ABI 14 grammars (TSLexMode) leave ReservedWordSetID zero. LexState 0xFFFF is the C (uint16_t)-1 marker and must be kept.

type Lexer 

type Lexer struct {
	// Current code point: 0 at end of input, -1 for invalid UTF-8.
	Lookahead int32
	// Set by lex functions (a symbol) and scanners (an external token index).
	ResultSymbol Symbol
	// contains filtered or unexported fields
}

Lexer is the C Lexer with its embedded TSLexer. Generated lex functions and scanners use only the two exported fields and the exported methods.

func (*Lexer) Advance 

func (l *Lexer) Advance(skip bool)

ts_lexer__advance, with ts_lexer__do_advance folded in: the lex functions call this once per character, and one call is cheaper than two.

func (*Lexer) AdvanceRun 

func (l *Lexer) AdvanceRun(set *[256]bool, skip bool)

AdvanceRun is Advance(skip) repeated for as long as Lookahead is an ASCII character in set. It has no C counterpart: generated lex functions call it in states that loop on themselves (cmd/gengrammar/run.go). set[0] and set[128:] are false.

func (*Lexer) EOF 

func (l *Lexer) EOF() bool

ts_lexer__eof

func (*Lexer) GetColumn 

func (l *Lexer) GetColumn() uint32

ts_lexer__get_column

func (*Lexer) IsAtIncludedRangeStart 

func (l *Lexer) IsAtIncludedRangeStart() bool

ts_lexer__is_at_included_range_start

func (*Lexer) MarkEnd 

func (l *Lexer) MarkEnd()

ts_lexer__mark_end

type LogType 

type LogType uint8

TSLogType. 

const (
	LogTypeParse LogType = 0
	LogTypeLex   LogType = 1
)

type Logger 

type Logger func(t LogType, msg string)

Logger receives the same messages the C runtime passes to TSLogger.log.

type MapSlice 

type MapSlice struct {
	Index  uint16
	Length uint16
}

TSMapSlice. Indexes the field and supertype maps.

type Node 

type Node struct {
	// contains filtered or unexported fields
}

TSNode, packed into four fields so the compiler keeps it in registers. A small comparable value; the zero Node is the null node. id is the slot of the subtree in the tree's children slab, where C has a const Subtree *.

func (Node) Child 

func (n Node) Child(i uint32) Node

ts_node_child

func (Node) ChildByFieldID 

func (n Node) ChildByFieldID(fieldID FieldID) Node

ts_node_child_by_field_id

func (Node) ChildByFieldName 

func (n Node) ChildByFieldName(name string) Node

ts_node_child_by_field_name

func (Node) ChildCount 

func (n Node) ChildCount() uint32

ts_node_child_count

func (Node) Children 

func (n Node) Children(buf []Node) []Node

Children appends the children of n to buf: the nodes Child(0) to Child(ChildCount()-1), found in one pass. ts_node_child rescans from the first child on every call, so a Child(i) loop is quadratic. No C counterpart.

func (Node) ChildrenByFieldName 

func (n Node) ChildrenByFieldName(name string, c *TreeCursor) []Node

Same loop as go-tree-sitter's Node.ChildrenByFieldName.

func (Node) EndByte 

func (n Node) EndByte() uint32

ts_node_end_byte

func (Node) EndPosition 

func (n Node) EndPosition() Point

ts_node_end_point

func (Node) Equals 

func (n Node) Equals(other Node) bool

ts_node_eq

func (Node) FieldNameForChild 

func (n Node) FieldNameForChild(childIndex uint32) string

ts_node_field_name_for_child. Returns "" where C returns NULL.

func (Node) HasError 

func (n Node) HasError() bool

ts_node_has_error

func (Node) IsError 

func (n Node) IsError() bool

ts_node_is_error

func (Node) IsExtra 

func (n Node) IsExtra() bool

ts_node_is_extra

func (Node) IsMissing 

func (n Node) IsMissing() bool

ts_node_is_missing

func (Node) IsNamed 

func (n Node) IsNamed() bool

ts_node_is_named

func (Node) IsNull 

func (n Node) IsNull() bool

ts_node_is_null

func (Node) Kind 

func (n Node) Kind() string

ts_node_type

func (Node) Language 

func (n Node) Language() *Language

ts_node_language

func (Node) NamedChild 

func (n Node) NamedChild(i uint32) Node

ts_node_named_child

func (Node) NamedChildCount 

func (n Node) NamedChildCount() uint32

ts_node_named_child_count

func (Node) NamedChildren 

func (n Node) NamedChildren(buf []Node) []Node

NamedChildren is Children for NamedChild.

func (Node) Parent 

func (n Node) Parent() Node

ts_node_parent

func (Node) Preorder 

func (n Node) Preorder(fn func(Node) bool)

Preorder calls fn for n and, where fn returns true, for the nodes below it in document order: the walk a recursive Children loop makes, with no slice per node. No C counterpart.

func (Node) PreorderMask 

func (n Node) PreorderMask(mask uint32, fn func(Node) bool)

PreorderMask is Preorder for an fn that returns true and does nothing else on a node whose kind is not in mask (see Language.KindMask): fn is called for the nodes of those kinds only, and a subtree that holds none of them is not entered. No C counterpart.

func (Node) StartByte 

func (n Node) StartByte() uint32

ts_node_start_byte

func (Node) StartPosition 

func (n Node) StartPosition() Point

ts_node_start_point

func (Node) String 

func (n Node) String() string

ts_node_string

func (Node) Symbol 

func (n Node) Symbol() Symbol

ts_node_symbol

func (Node) Utf8Text 

func (n Node) Utf8Text(src []byte) string

string(src[StartByte:EndByte]), as go-tree-sitter does.

func (Node) Walk 

func (n Node) Walk() *TreeCursor

ts_tree_cursor_new

type ParseAction 

type ParseAction struct {
	Type              uint8
	ChildCount        uint8
	Symbol            Symbol
	DynamicPrecedence int16
	ProductionID      uint16
}

ParseAction is one 8-byte, pointer-free slot of Language.ParseActions. It stands for both C unions, TSParseActionEntry and TSParseAction, and keeps the C byte layout. Which variant a slot holds follows from its position: a header slot is followed by Count() action slots. 

slot      Type   ChildCount   Symbol   DynamicPrecedence      ProductionID
header    count  reusable     0        0                      0
shift     0      0            state    bit0 extra, bit1 rep.  0
reduce    1      child_count  symbol   dynamic_precedence     production_id
accept    2      0            0        0                      0
recover   3      0            0        0                      0

The fields are named for the reduce variant; use the methods for the rest.

func (ParseAction) Count 

func (a ParseAction) Count() uint8

entry.count of a header slot.

func (ParseAction) Extra 

func (a ParseAction) Extra() bool

shift.extra

func (ParseAction) Repetition 

func (a ParseAction) Repetition() bool

shift.repetition

func (ParseAction) Reusable 

func (a ParseAction) Reusable() bool

entry.reusable of a header slot.

func (ParseAction) State 

func (a ParseAction) State() StateID

shift.state

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

TSParser. Not safe for concurrent use; one per goroutine.

func NewParser 

func NewParser() *Parser

ts_parser_new

func (*Parser) Parse 

func (p *Parser) Parse(src []byte) *Tree

ts_parser_parse with old_tree NULL and UTF-8 input. Returns nil if no language is set. The tree is valid until the next Parse, Reset or SetLanguage on p.

func (*Parser) ParseBudget 

func (p *Parser) ParseBudget(src []byte, maxStall uint32) *Tree

ParseBudget is Parse, but returns nil once advance has been called more than maxStall times in a row without passing the furthest byte reached. 0 is no limit. The count depends only on the language and src.

func (*Parser) Reset 

func (p *Parser) Reset()

ts_parser_reset. C releases the stack, the token cache and the finished tree one reference at a time. Here everything lives in the parser's slabs and nothing outlives the reset, so they are truncated instead. The tree of the last Parse dies with them.

func (*Parser) SetLanguage 

func (p *Parser) SetLanguage(l *Language) error

ts_parser_set_language. Errors on an incompatible ABIVersion.

func (*Parser) SetLogger 

func (p *Parser) SetLogger(fn Logger)

ts_parser_set_logger. nil turns logging off.

type Point 

type Point struct {
	Row    uint32
	Column uint32
}

TSPoint. Column counts bytes.

type StateID 

type StateID uint16

TSSymbol, TSStateId, TSFieldId.

type Symbol 

type Symbol uint16

TSSymbol, TSStateId, TSFieldId.

type SymbolMetadata 

type SymbolMetadata struct {
	Visible   bool
	Named     bool
	Supertype bool
}

TSSymbolMetadata.

type Tree 

type Tree struct {
	// contains filtered or unexported fields
}

TSTree. It borrows the slabs of the parser that made it.

func (*Tree) Close 

func (t *Tree) Close()

ts_tree_delete. The memory belongs to the parser, which reuses it on its next Parse, so there is nothing to do.

func (*Tree) Language 

func (t *Tree) Language() *Language

ts_tree_language

func (*Tree) RootNode 

func (t *Tree) RootNode() Node

ts_tree_root_node

type TreeCursor 

type TreeCursor struct {
	// contains filtered or unexported fields
}

TSTreeCursor.

func (*TreeCursor) Close 

func (c *TreeCursor) Close()

ts_tree_cursor_delete

func (*TreeCursor) FieldID 

func (c *TreeCursor) FieldID() FieldID

ts_tree_cursor_current_field_id

func (*TreeCursor) FieldName 

func (c *TreeCursor) FieldName() string

ts_tree_cursor_current_field_name

func (*TreeCursor) GotoFirstChild 

func (c *TreeCursor) GotoFirstChild() bool

ts_tree_cursor_goto_first_child, with the internal variant and the child_count guard of ts_tree_cursor_iterate_children folded in.

func (*TreeCursor) GotoNextSibling 

func (c *TreeCursor) GotoNextSibling() bool

ts_tree_cursor_goto_next_sibling

func (*TreeCursor) GotoParent 

func (c *TreeCursor) GotoParent() bool

ts_tree_cursor_goto_parent

func (*TreeCursor) Node 

func (c *TreeCursor) Node() Node

ts_tree_cursor_current_node, alias read from the entry.

func (*TreeCursor) Reset 

func (c *TreeCursor) Reset(n Node)

ts_tree_cursor_reset

Source Files

View all Source files

Directories

Path Synopsis
cmd
gengrammar command
Command gengrammar turns a tree-sitter parser.c into a Go package for gositter: tables.go, lex.go and language.go.
 Command gengrammar turns a tree-sitter parser.c into a Go package for gositter: tables.go, lex.go and language.go.
grammars
bash
c
cpp
csharp
golang
hcl
internal/cchar
Package cchar gives scanners the plain char of the C compiler that the matching cgo build uses, so a port can follow C where a scanner stores a code point in a char and reads it back.
 Package cchar gives scanners the plain char of the C compiler that the matching cgo build uses, so a port can follow C where a scanner stores a code point in a char and reads it back.
java
javascript
json
julia
kotlin
lua
php
python
ruby
rust
scala
tsx
typescript
verilog
zig
internal
treedump

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.