grammars

package
v0.38.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2026 License: MIT Imports: 30 Imported by: 0

Documentation

Overview

c_sharp precise ExternalLexStates — DEFAULT since the 2026-07 cliff campaign (previously staged behind -tags csharp_precise_els).

The embedded c_sharp grammar blob preserves LexModes[state].ExternalLexState (histogram matches C exactly: 10 distinct states) but shipped an EMPTY ExternalLexStates validity table (ts2go dropped it). With the table empty, dfaTokenSource (parser_dfa_token_source.go) cannot use its precise per-state external path and instead unions externalValidMaskByState across all active GLR stacks. Under multi-stack pressure that union over-approximates external-token validity and corrupts the STATEFUL interpolated-string scanner (grammars/csharp_scanner.go): at an interpolation hole's close it reported external-lex-state 2 (expression start) instead of 5 (string content), the scanner's interpolation stack never popped (measured: 27 pushes, 0 pops), and pass-1 on DeclaredTypeManager.cs died no_stacks_alive at byte 31888 behind a ~1,000-pass retry cascade.

The table below is copied VERBATIM from tree-sitter-c-sharp src/parser.c (ts_external_scanner_states[10][12], pinned commit in languages.lock). Registering it (a) restores the precise per-state external path, and (b) satisfies the C-recovery election gate (DiagnoseCRecoveryGate), so c_sharp runs the faithful C error-recovery cost competition by default.

The two engine blockers that kept this staged were fixed in the same campaign:

  • parser_recover_c.go cCondenseAndResume: the C MAX_VERSION_COUNT window now bounds DISTINCT merge keys (C merges same-key versions via ts_stack_merge before its cap; the old raw positional trim dropped whole grammar interpretations).
  • parser.go usesPrimaryExternalScannerStateForGLR: the c_sharp primary-stack-only pin now applies only while ExternalLexStates is empty; with the precise table the scored per-row path sees all stack states, so interpolation-hole stacks receive interpolation_close_brace.

Validation: grammars/csharp_external_lex_states_regression_test.go.

Verbatim ts_external_scanner_states[10][8] from the pinned tree-sitter-javascript parser.c. Ordinary scanner arbitration needs these rows to distinguish ASI-valid states from expression-continuation states; action-table fallback is broader and can reverse ASI/comment ordering.

WHY PRECISE ELS IS DEFAULT BUT C RECOVERY IS NOT (2026-07 cliff campaign measurement) Unlike c_sharp — whose stateful interpolation scanner was actively corrupted by the union-mask fallback, so the precise table + election fixed real misparses (DeclaredTypeManager.cs 31 ERROR nodes -> 0) — javascript's worst perf_scan sweep files parse with ZERO error nodes either way, so the table has nothing to fix there today, and ELECTION is a measured regression on that slice (loaded-box interleaved A/B, elected vs GOT_C_RECOVERY=0, live engine with both campaign engine fixes): 

deps/undici/undici.js               2.2-2.4s -> ~5.1s   (accepted both)
deps/v8/.../unicode-test.js         ~370ms   -> ~600ms  (median of 4+4)
deps/v8/.../box2d.js                memory_budget @49543 either way
deps/v8/.../lua_binarytrees.js      memory_budget @6745 either way
deps/amaro/dist/index.js            ~100-135ms either way

The elected path's per-token cost (cRecoveryEnabled lexing via NextWithErrorRuns + acquire-token bookkeeping) is paid on CLEAN parses, and javascript's actual sweep cliffs (asm.js megafunctions dying on memory_budget) are unrelated to external-scanner validity. JavaScript is therefore explicitly opted out of default C-recovery election even though precise scanner validity is now always enabled. Re-enable recovery only after the memory-budget cliff is fixed and a sweep shows it at worst neutral.

Columns are the JavaScript external token indices, in the language's ExternalSymbols order (== C enum order): 

0 _automatic_semicolon  4 ||
1 _template_chars       5 escape_sequence
2 _ternary_qmark        6 regex_pattern
3 html_comment          7 jsx_text

Source: https://github.com/tree-sitter/tree-sitter-javascript 58404d8cf191d69f2674a8fd507bd5776f46cb11 src/parser.c

Package grammars provides built-in and extension tree-sitter grammars with lazy loading. Most built-in grammars are currently shipped as compressed ts2go blobs, while extension grammars can come from grammargen-generated loaders. Use AllLanguages to enumerate available grammars, DetectLanguage to match by file extension or shebang, or call individual language functions (e.g. GoLanguage()) for direct access.

Index

Constants

This section is empty.

Variables

View Source 
var ParseSmokeSamples = map[string]string{}/* 206 elements not displayed */

ParseSmokeSamples maps language names to minimal source snippets used for smoke testing. Each sample should parse without errors under the language's supported backend.

Languages not listed here fall back to "x\n" as a generic probe.

Functions

func AdaLanguage 

func AdaLanguage() *gotreesitter.Language

AdaLanguage returns the ada language definition.

func AdaptScannerForLanguage 

func AdaptScannerForLanguage(name string, targetLang *gotreesitter.Language) bool

AdaptScannerForLanguage adapts the registered hand-written scanner for the named language to work with a different Language (e.g., one produced by grammargen). It loads the ts2go reference Language to get the scanner's native Symbol IDs, then builds an adapter that remaps them to the target Language's Symbol IDs.

func AgdaLanguage 

func AgdaLanguage() *gotreesitter.Language

AgdaLanguage returns the agda language definition.

func AngularLanguage 

func AngularLanguage() *gotreesitter.Language

AngularLanguage returns the angular language definition.

func ApexLanguage 

func ApexLanguage() *gotreesitter.Language

ApexLanguage returns the apex language definition.

func ArduinoLanguage 

func ArduinoLanguage() *gotreesitter.Language

ArduinoLanguage returns the arduino language definition.

func AsmLanguage 

func AsmLanguage() *gotreesitter.Language

AsmLanguage returns the asm language definition.

func AstroLanguage 

func AstroLanguage() *gotreesitter.Language

AstroLanguage returns the astro language definition.

func AttachLanguageSupport 

func AttachLanguageSupport(name string, lang *gotreesitter.Language) bool

AttachLanguageSupport attaches registered scanner and external lex-state support for name to lang. Certified runtime profiles require the original blob identity and are attached by LoadLanguage or the embedded loader.

func AuthzedLanguage 

func AuthzedLanguage() *gotreesitter.Language

AuthzedLanguage returns the authzed language definition.

func AwkLanguage 

func AwkLanguage() *gotreesitter.Language

AwkLanguage returns the awk language definition.

func BashLanguage 

func BashLanguage() *gotreesitter.Language

BashLanguage returns the bash language definition.

func BassLanguage 

func BassLanguage() *gotreesitter.Language

BassLanguage returns the bass language definition.

func BeancountLanguage 

func BeancountLanguage() *gotreesitter.Language

BeancountLanguage returns the beancount language definition.

func BibtexLanguage 

func BibtexLanguage() *gotreesitter.Language

BibtexLanguage returns the bibtex language definition.

func BicepLanguage 

func BicepLanguage() *gotreesitter.Language

BicepLanguage returns the bicep language definition.

func BitbakeLanguage 

func BitbakeLanguage() *gotreesitter.Language

BitbakeLanguage returns the bitbake language definition.

func BladeLanguage 

func BladeLanguage() *gotreesitter.Language

BladeLanguage returns the blade language definition.

func BlobByName 

func BlobByName(name string) []byte

BlobByName returns the raw grammar blob for the named language (e.g. "go", "python"). Returns nil if the language blob is not found. Returned bytes are either legacy gzip+gob data or a versioned language-blob envelope. Consumers must load them with gotreesitter.LoadLanguage (or the equivalent grammars loader) rather than assuming a gzip header. The bytes remain suitable for browser-side WASM modules that use that runtime decoder.

func BrightscriptLanguage 

func BrightscriptLanguage() *gotreesitter.Language

BrightscriptLanguage returns the brightscript language definition.

func CLanguage 

func CLanguage() *gotreesitter.Language

CLanguage returns the c language definition.

func CSharpLanguage 

func CSharpLanguage() *gotreesitter.Language

CSharpLanguage returns the c_sharp language definition.

func CaddyLanguage 

func CaddyLanguage() *gotreesitter.Language

CaddyLanguage returns the caddy language definition.

func CairoLanguage 

func CairoLanguage() *gotreesitter.Language

CairoLanguage returns the cairo language definition.

func CapnpLanguage 

func CapnpLanguage() *gotreesitter.Language

CapnpLanguage returns the capnp language definition.

func ChatitoLanguage 

func ChatitoLanguage() *gotreesitter.Language

ChatitoLanguage returns the chatito language definition.

func CircomLanguage 

func CircomLanguage() *gotreesitter.Language

CircomLanguage returns the circom language definition.

func ClojureLanguage 

func ClojureLanguage() *gotreesitter.Language

ClojureLanguage returns the clojure language definition.

func CmakeLanguage 

func CmakeLanguage() *gotreesitter.Language

CmakeLanguage returns the cmake language definition.

func CobolLanguage 

func CobolLanguage() *gotreesitter.Language

CobolLanguage returns the cobol language definition.

func CommentLanguage 

func CommentLanguage() *gotreesitter.Language

CommentLanguage returns the comment language definition.

func CommonlispLanguage 

func CommonlispLanguage() *gotreesitter.Language

CommonlispLanguage returns the commonlisp language definition.

func CooklangLanguage 

func CooklangLanguage() *gotreesitter.Language

CooklangLanguage returns the cooklang language definition.

func Core100LanguageNames 

func Core100LanguageNames() []string

Core100LanguageNames returns a copy of the compile-time Core100 grammar set.

func CornLanguage 

func CornLanguage() *gotreesitter.Language

CornLanguage returns the corn language definition.

func CponLanguage 

func CponLanguage() *gotreesitter.Language

CponLanguage returns the cpon language definition.

func CppLanguage 

func CppLanguage() *gotreesitter.Language

CppLanguage returns the cpp language definition.

func CrystalLanguage 

func CrystalLanguage() *gotreesitter.Language

CrystalLanguage returns the crystal language definition.

func CssLanguage 

func CssLanguage() *gotreesitter.Language

CssLanguage returns the css language definition.

func CsvLanguage 

func CsvLanguage() *gotreesitter.Language

CsvLanguage returns the csv language definition.

func CudaLanguage 

func CudaLanguage() *gotreesitter.Language

CudaLanguage returns the cuda language definition.

func CueLanguage 

func CueLanguage() *gotreesitter.Language

CueLanguage returns the cue language definition.

func CylcLanguage 

func CylcLanguage() *gotreesitter.Language

CylcLanguage returns the cylc language definition.

func DLanguage 

func DLanguage() *gotreesitter.Language

DLanguage returns the d language definition.

func DartLanguage 

func DartLanguage() *gotreesitter.Language

DartLanguage returns the dart language definition.

func DesktopLanguage 

func DesktopLanguage() *gotreesitter.Language

DesktopLanguage returns the desktop language definition.

func DevicetreeLanguage 

func DevicetreeLanguage() *gotreesitter.Language

DevicetreeLanguage returns the devicetree language definition.

func DhallLanguage 

func DhallLanguage() *gotreesitter.Language

DhallLanguage returns the dhall language definition.

func DiffLanguage 

func DiffLanguage() *gotreesitter.Language

DiffLanguage returns the diff language definition.

func DisassemblyLanguage 

func DisassemblyLanguage() *gotreesitter.Language

DisassemblyLanguage returns the disassembly language definition.

func DisplayName 

func DisplayName(entry *LangEntry) string

DisplayName returns the linguist canonical display name for a language (e.g., "C++" for cpp, "JavaScript" for javascript). Falls back to title-casing the grammar name if no linguist match exists.

func DjotLanguage 

func DjotLanguage() *gotreesitter.Language

DjotLanguage returns the djot language definition.

func DockerfileLanguage 

func DockerfileLanguage() *gotreesitter.Language

DockerfileLanguage returns the dockerfile language definition.

func DotLanguage 

func DotLanguage() *gotreesitter.Language

DotLanguage returns the dot language definition.

func DoxygenLanguage 

func DoxygenLanguage() *gotreesitter.Language

DoxygenLanguage returns the doxygen language definition.

func DtdLanguage 

func DtdLanguage() *gotreesitter.Language

DtdLanguage returns the dtd language definition.

func EarthfileLanguage 

func EarthfileLanguage() *gotreesitter.Language

EarthfileLanguage returns the earthfile language definition.

func EbnfLanguage 

func EbnfLanguage() *gotreesitter.Language

EbnfLanguage returns the ebnf language definition.

func EditorconfigLanguage 

func EditorconfigLanguage() *gotreesitter.Language

EditorconfigLanguage returns the editorconfig language definition.

func EdsLanguage 

func EdsLanguage() *gotreesitter.Language

EdsLanguage returns the eds language definition.

func EexLanguage 

func EexLanguage() *gotreesitter.Language

EexLanguage returns the eex language definition.

func ElispLanguage 

func ElispLanguage() *gotreesitter.Language

ElispLanguage returns the elisp language definition.

func ElixirLanguage 

func ElixirLanguage() *gotreesitter.Language

ElixirLanguage returns the elixir language definition.

func ElmLanguage 

func ElmLanguage() *gotreesitter.Language

ElmLanguage returns the elm language definition.

func ElsaLanguage 

func ElsaLanguage() *gotreesitter.Language

ElsaLanguage returns the elsa language definition.

func EmbeddedLanguageCacheStats 

func EmbeddedLanguageCacheStats() (loaded int, limit int)

EmbeddedLanguageCacheStats returns the current decoded-grammar cache size and configured cache limit.

func EmbeddedLanguageIdleConfig 

func EmbeddedLanguageIdleConfig() (ttl time.Duration, sweepInterval time.Duration)

EmbeddedLanguageIdleConfig returns the current idle eviction settings.

func EmbeddedTemplateLanguage 

func EmbeddedTemplateLanguage() *gotreesitter.Language

EmbeddedTemplateLanguage returns the embedded_template language definition.

func EnforceLanguage 

func EnforceLanguage() *gotreesitter.Language

EnforceLanguage returns the enforce language definition.

func ErlangLanguage 

func ErlangLanguage() *gotreesitter.Language

ErlangLanguage returns the erlang language definition.

func FacilityLanguage 

func FacilityLanguage() *gotreesitter.Language

FacilityLanguage returns the facility language definition.

func FaustLanguage 

func FaustLanguage() *gotreesitter.Language

FaustLanguage returns the faust language definition.

func FennelLanguage 

func FennelLanguage() *gotreesitter.Language

FennelLanguage returns the fennel language definition.

func FidlLanguage 

func FidlLanguage() *gotreesitter.Language

FidlLanguage returns the fidl language definition.

func FirrtlLanguage 

func FirrtlLanguage() *gotreesitter.Language

FirrtlLanguage returns the firrtl language definition.

func FishLanguage 

func FishLanguage() *gotreesitter.Language

FishLanguage returns the fish language definition.

func FoamLanguage 

func FoamLanguage() *gotreesitter.Language

FoamLanguage returns the foam language definition.

func ForthLanguage 

func ForthLanguage() *gotreesitter.Language

ForthLanguage returns the forth language definition.

func FortranLanguage 

func FortranLanguage() *gotreesitter.Language

FortranLanguage returns the fortran language definition.

func FsharpLanguage 

func FsharpLanguage() *gotreesitter.Language

FsharpLanguage returns the fsharp language definition.

func GdscriptLanguage 

func GdscriptLanguage() *gotreesitter.Language

GdscriptLanguage returns the gdscript language definition.

func GitConfigLanguage 

func GitConfigLanguage() *gotreesitter.Language

GitConfigLanguage returns the git_config language definition.

func GitRebaseLanguage 

func GitRebaseLanguage() *gotreesitter.Language

GitRebaseLanguage returns the git_rebase language definition.

func GitattributesLanguage 

func GitattributesLanguage() *gotreesitter.Language

GitattributesLanguage returns the gitattributes language definition.

func GitcommitLanguage 

func GitcommitLanguage() *gotreesitter.Language

GitcommitLanguage returns the gitcommit language definition.

func GitignoreLanguage 

func GitignoreLanguage() *gotreesitter.Language

GitignoreLanguage returns the gitignore language definition.

func GleamLanguage 

func GleamLanguage() *gotreesitter.Language

GleamLanguage returns the gleam language definition.

func GlslLanguage 

func GlslLanguage() *gotreesitter.Language

GlslLanguage returns the glsl language definition.

func GnLanguage 

func GnLanguage() *gotreesitter.Language

GnLanguage returns the gn language definition.

func GoLanguage 

func GoLanguage() *gotreesitter.Language

GoLanguage returns the go language definition.

func GodotResourceLanguage 

func GodotResourceLanguage() *gotreesitter.Language

GodotResourceLanguage returns the godot_resource language definition.

func GomodLanguage 

func GomodLanguage() *gotreesitter.Language

GomodLanguage returns the gomod language definition.

func GraphqlLanguage 

func GraphqlLanguage() *gotreesitter.Language

GraphqlLanguage returns the graphql language definition.

func GroovyLanguage 

func GroovyLanguage() *gotreesitter.Language

GroovyLanguage returns the groovy language definition.

func HackLanguage 

func HackLanguage() *gotreesitter.Language

HackLanguage returns the hack language definition.

func HareLanguage 

func HareLanguage() *gotreesitter.Language

HareLanguage returns the hare language definition.

func HaskellLanguage 

func HaskellLanguage() *gotreesitter.Language

HaskellLanguage returns the haskell language definition.

func HaxeLanguage 

func HaxeLanguage() *gotreesitter.Language

HaxeLanguage returns the haxe language definition.

func HclLanguage 

func HclLanguage() *gotreesitter.Language

HclLanguage returns the hcl language definition.

func HeexLanguage 

func HeexLanguage() *gotreesitter.Language

HeexLanguage returns the heex language definition.

func HlslLanguage 

func HlslLanguage() *gotreesitter.Language

HlslLanguage returns the hlsl language definition.

func HtmlLanguage 

func HtmlLanguage() *gotreesitter.Language

HtmlLanguage returns the html language definition.

func HttpLanguage 

func HttpLanguage() *gotreesitter.Language

HttpLanguage returns the http language definition.

func HurlLanguage 

func HurlLanguage() *gotreesitter.Language

HurlLanguage returns the hurl language definition.

func HyprlangLanguage 

func HyprlangLanguage() *gotreesitter.Language

HyprlangLanguage returns the hyprlang language definition.

func IniLanguage 

func IniLanguage() *gotreesitter.Language

IniLanguage returns the ini language definition.

func JanetLanguage 

func JanetLanguage() *gotreesitter.Language

JanetLanguage returns the janet language definition.

func JavaLanguage 

func JavaLanguage() *gotreesitter.Language

JavaLanguage returns the java language definition.

func JavascriptLanguage 

func JavascriptLanguage() *gotreesitter.Language

JavascriptLanguage returns the javascript language definition.

func Jinja2Language 

func Jinja2Language() *gotreesitter.Language

Jinja2Language returns the jinja2 language definition.

func JqLanguage 

func JqLanguage() *gotreesitter.Language

JqLanguage returns the jq language definition.

func JsdocLanguage 

func JsdocLanguage() *gotreesitter.Language

JsdocLanguage returns the jsdoc language definition.

func Json5Language 

func Json5Language() *gotreesitter.Language

Json5Language returns the json5 language definition.

func JsonLanguage 

func JsonLanguage() *gotreesitter.Language

JsonLanguage returns the json language definition.

func JsonnetLanguage 

func JsonnetLanguage() *gotreesitter.Language

JsonnetLanguage returns the jsonnet language definition.

func JuliaLanguage 

func JuliaLanguage() *gotreesitter.Language

JuliaLanguage returns the julia language definition.

func JustLanguage 

func JustLanguage() *gotreesitter.Language

JustLanguage returns the just language definition.

func KconfigLanguage 

func KconfigLanguage() *gotreesitter.Language

KconfigLanguage returns the kconfig language definition.

func KdlLanguage 

func KdlLanguage() *gotreesitter.Language

KdlLanguage returns the kdl language definition.

func KotlinLanguage 

func KotlinLanguage() *gotreesitter.Language

KotlinLanguage returns the kotlin language definition.

func LedgerLanguage 

func LedgerLanguage() *gotreesitter.Language

LedgerLanguage returns the ledger language definition.

func LessLanguage 

func LessLanguage() *gotreesitter.Language

LessLanguage returns the less language definition.

func LinkerscriptLanguage 

func LinkerscriptLanguage() *gotreesitter.Language

LinkerscriptLanguage returns the linkerscript language definition.

func LiquidLanguage 

func LiquidLanguage() *gotreesitter.Language

LiquidLanguage returns the liquid language definition.

func LlvmLanguage 

func LlvmLanguage() *gotreesitter.Language

LlvmLanguage returns the llvm language definition.

func LoadLanguage 

func LoadLanguage(name string, data []byte) (*gotreesitter.Language, error)

LoadLanguage deserializes a raw grammar blob and attaches registered gotreesitter/grammars runtime support for name, including external scanners, external lex-state tables, and any runtime profile certified for this exact blob. Use this instead of gotreesitter.LoadLanguage when loading blobs that came from BlobByName, grammar_subset, or a remote WASM bundle and the caller knows the language name.

func LookupExternalLexStates 

func LookupExternalLexStates(name string) [][]bool

LookupExternalLexStates returns the registered external lex states table for the given language name, or nil if none is registered.

func LookupExternalScanner 

func LookupExternalScanner(name string) gotreesitter.ExternalScanner

LookupExternalScanner returns the registered hand-written external scanner for the given language name (e.g. "python"), or nil if none is registered.

func LuaLanguage 

func LuaLanguage() *gotreesitter.Language

LuaLanguage returns the lua language definition.

func LuauLanguage 

func LuauLanguage() *gotreesitter.Language

LuauLanguage returns the luau language definition.

func MakeLanguage 

func MakeLanguage() *gotreesitter.Language

MakeLanguage returns the make language definition.

func MarkdownInlineLanguage 

func MarkdownInlineLanguage() *gotreesitter.Language

MarkdownInlineLanguage returns the markdown_inline language definition.

func MarkdownLanguage 

func MarkdownLanguage() *gotreesitter.Language

MarkdownLanguage returns the markdown language definition.

func MatlabLanguage 

func MatlabLanguage() *gotreesitter.Language

MatlabLanguage returns the matlab language definition.

func MermaidLanguage 

func MermaidLanguage() *gotreesitter.Language

MermaidLanguage returns the mermaid language definition.

func MesonLanguage 

func MesonLanguage() *gotreesitter.Language

MesonLanguage returns the meson language definition.

func MojoLanguage 

func MojoLanguage() *gotreesitter.Language

MojoLanguage returns the mojo language definition.

func MoveLanguage 

func MoveLanguage() *gotreesitter.Language

MoveLanguage returns the move language definition.

func NewAuthzedTokenSourceOrEOF 

func NewAuthzedTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewAuthzedTokenSourceOrEOF returns an authzed token source, or EOF-only fallback if symbol setup fails.

func NewCTokenSourceOrEOF 

func NewCTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewCTokenSourceOrEOF returns a C token source, or EOF-only fallback if symbol setup fails.

func NewGenericTokenSourceOrEOF 

func NewGenericTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewGenericTokenSourceOrEOF returns a generic token source, or EOF-only fallback if setup fails.

func NewGoTokenSourceOrEOF 

func NewGoTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewGoTokenSourceOrEOF returns a token source for callers that cannot surface constructor errors through their own API.

func NewHTMLTokenSourceOrEOF 

func NewHTMLTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewHTMLTokenSourceOrEOF returns an HTML token source, or EOF-only fallback if setup fails.

func NewJSONTokenSourceOrEOF 

func NewJSONTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewJSONTokenSourceOrEOF returns a token source for callers that cannot surface constructor errors through their API.

func NewJavaTokenSourceOrEOF 

func NewJavaTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewJavaTokenSourceOrEOF returns a Java token source, or EOF-only fallback if symbol setup fails.

func NewLuaTokenSourceOrEOF 

func NewLuaTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewLuaTokenSourceOrEOF returns a Lua token source, or EOF-only fallback if symbol setup fails.

func NewTomlTokenSourceOrEOF 

func NewTomlTokenSourceOrEOF(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource

NewTomlTokenSourceOrEOF returns a TOML token source, or EOF-only fallback if setup fails.

func NginxLanguage 

func NginxLanguage() *gotreesitter.Language

NginxLanguage returns the nginx language definition.

func NickelLanguage 

func NickelLanguage() *gotreesitter.Language

NickelLanguage returns the nickel language definition.

func NimLanguage 

func NimLanguage() *gotreesitter.Language

NimLanguage returns the nim language definition.

func NinjaLanguage 

func NinjaLanguage() *gotreesitter.Language

NinjaLanguage returns the ninja language definition.

func NixLanguage 

func NixLanguage() *gotreesitter.Language

NixLanguage returns the nix language definition.

func NorgLanguage 

func NorgLanguage() *gotreesitter.Language

NorgLanguage returns the norg language definition.

func NushellLanguage 

func NushellLanguage() *gotreesitter.Language

NushellLanguage returns the nushell language definition.

func ObjcLanguage 

func ObjcLanguage() *gotreesitter.Language

ObjcLanguage returns the objc language definition.

func OcamlLanguage 

func OcamlLanguage() *gotreesitter.Language

OcamlLanguage returns the ocaml language definition.

func OdinLanguage 

func OdinLanguage() *gotreesitter.Language

OdinLanguage returns the odin language definition.

func OrgLanguage 

func OrgLanguage() *gotreesitter.Language

OrgLanguage returns the org language definition.

func ParseFile 

func ParseFile(filename string, source []byte) (*gotreesitter.BoundTree, error)

ParseFile detects the language from filename, parses source, and returns a BoundTree. The caller must call Release() on the returned BoundTree.

Detection is content-aware (DetectLanguageWithContent): a plain ".h" file whose content looks like C++ (templates, namespaces, std::, ...) is parsed with the C++ grammar rather than the extension-only default of C.

func ParseFilePooled 

func ParseFilePooled(filename string, source []byte) (*gotreesitter.BoundTree, error)

ParseFilePooled is like ParseFile but reuses a per-language ParserPool to avoid allocating a new parser on every call. It is safe for concurrent use. The caller must call Release() on the returned BoundTree.

Detection is content-aware; see the ParseFile doc comment.

func ParseSmokeSample 

func ParseSmokeSample(name string) string

ParseSmokeSample returns the smoke test sample for the given language, falling back to "x\n" for languages without a dedicated sample.

func PascalLanguage 

func PascalLanguage() *gotreesitter.Language

PascalLanguage returns the pascal language definition.

func PemLanguage 

func PemLanguage() *gotreesitter.Language

PemLanguage returns the pem language definition.

func PerlLanguage 

func PerlLanguage() *gotreesitter.Language

PerlLanguage returns the perl language definition.

func PhpLanguage 

func PhpLanguage() *gotreesitter.Language

PhpLanguage returns the php language definition.

func PklLanguage 

func PklLanguage() *gotreesitter.Language

PklLanguage returns the pkl language definition.

func PowershellLanguage 

func PowershellLanguage() *gotreesitter.Language

PowershellLanguage returns the powershell language definition.

func PrismaLanguage 

func PrismaLanguage() *gotreesitter.Language

PrismaLanguage returns the prisma language definition.

func PrologLanguage 

func PrologLanguage() *gotreesitter.Language

PrologLanguage returns the prolog language definition.

func PromqlLanguage 

func PromqlLanguage() *gotreesitter.Language

PromqlLanguage returns the promql language definition.

func PropertiesLanguage 

func PropertiesLanguage() *gotreesitter.Language

PropertiesLanguage returns the properties language definition.

func ProtoLanguage 

func ProtoLanguage() *gotreesitter.Language

ProtoLanguage returns the proto language definition.

func PugLanguage 

func PugLanguage() *gotreesitter.Language

PugLanguage returns the pug language definition.

func PuppetLanguage 

func PuppetLanguage() *gotreesitter.Language

PuppetLanguage returns the puppet language definition.

func PurescriptLanguage 

func PurescriptLanguage() *gotreesitter.Language

PurescriptLanguage returns the purescript language definition.

func PurgeEmbeddedLanguageCache 

func PurgeEmbeddedLanguageCache() int

PurgeEmbeddedLanguageCache removes all decoded grammar blobs from cache and returns the number of removed entries.

func PythonLanguage 

func PythonLanguage() *gotreesitter.Language

PythonLanguage returns the python language definition.

func QlLanguage 

func QlLanguage() *gotreesitter.Language

QlLanguage returns the ql language definition.

func RLanguage 

func RLanguage() *gotreesitter.Language

RLanguage returns the r language definition.

func RacketLanguage 

func RacketLanguage() *gotreesitter.Language

RacketLanguage returns the racket language definition.

func RegexLanguage 

func RegexLanguage() *gotreesitter.Language

RegexLanguage returns the regex language definition.

func Register 

func Register(entry LangEntry)

Register adds a language to the registry. If an entry with the same name already exists, it is replaced so that grammar updates take effect.

func RegisterExtension 

func RegisterExtension(ext ExtensionEntry)

RegisterExtension registers a grammar extension for file detection and markdown code fence highlighting.

func RegisterExternalLexStates 

func RegisterExternalLexStates(name string, states [][]bool)

RegisterExternalLexStates registers the external lex states table for a language, matching C tree-sitter's ts_external_scanner_states.

func RegisterExternalScanner 

func RegisterExternalScanner(name string, s gotreesitter.ExternalScanner)

RegisterExternalScanner registers an external scanner for a language name. This is called during init() by zzz_scanner_attachments.go.

func RegisterExternalScannerSpec 

func RegisterExternalScannerSpec(spec ExternalScannerSpec)

RegisterExternalScannerSpec records scanner-source metadata for a language.

func RegoLanguage 

func RegoLanguage() *gotreesitter.Language

RegoLanguage returns the rego language definition.

func RequirementsLanguage 

func RequirementsLanguage() *gotreesitter.Language

RequirementsLanguage returns the requirements language definition.

func RescriptLanguage 

func RescriptLanguage() *gotreesitter.Language

RescriptLanguage returns the rescript language definition.

func ResolveTagsQuery 

func ResolveTagsQuery(entry LangEntry) string

ResolveTagsQuery returns the tags query for a LangEntry, computing an inferred query from grammar symbols on first call if the entry lacks an explicit TagsQuery. The inferred result is cached by language name. This may trigger grammar loading for languages without an explicit TagsQuery.

func RobotLanguage 

func RobotLanguage() *gotreesitter.Language

RobotLanguage returns the robot language definition.

func RonLanguage 

func RonLanguage() *gotreesitter.Language

RonLanguage returns the ron language definition.

func RstLanguage 

func RstLanguage() *gotreesitter.Language

RstLanguage returns the rst language definition.

func RubyLanguage 

func RubyLanguage() *gotreesitter.Language

RubyLanguage returns the ruby language definition.

func RustLanguage 

func RustLanguage() *gotreesitter.Language

RustLanguage returns the rust language definition.

func ScalaLanguage 

func ScalaLanguage() *gotreesitter.Language

ScalaLanguage returns the scala language definition.

func SchemeLanguage 

func SchemeLanguage() *gotreesitter.Language

SchemeLanguage returns the scheme language definition.

func ScssLanguage 

func ScssLanguage() *gotreesitter.Language

ScssLanguage returns the scss language definition.

func SetEmbeddedLanguageCacheLimit 

func SetEmbeddedLanguageCacheLimit(limit int)

SetEmbeddedLanguageCacheLimit sets the maximum number of decoded grammar blobs retained in the in-process cache.

- limit < 0: unlimited cache size (default) - limit == 0: disable cache retention (decode on each call) - limit > 0: retain at most limit most recently used grammars

func SetEmbeddedLanguageIdleSweepInterval 

func SetEmbeddedLanguageIdleSweepInterval(interval time.Duration)

SetEmbeddedLanguageIdleSweepInterval controls how often idle cache entries are checked when idle eviction is enabled.

func SetEmbeddedLanguageIdleTTL 

func SetEmbeddedLanguageIdleTTL(ttl time.Duration)

SetEmbeddedLanguageIdleTTL controls idle-time eviction for decoded grammars. A value <= 0 disables idle eviction.

func SmithyLanguage 

func SmithyLanguage() *gotreesitter.Language

SmithyLanguage returns the smithy language definition.

func SolidityLanguage 

func SolidityLanguage() *gotreesitter.Language

SolidityLanguage returns the solidity language definition.

func SparqlLanguage 

func SparqlLanguage() *gotreesitter.Language

SparqlLanguage returns the sparql language definition.

func SqlLanguage 

func SqlLanguage() *gotreesitter.Language

SqlLanguage returns the sql language definition.

func SquirrelLanguage 

func SquirrelLanguage() *gotreesitter.Language

SquirrelLanguage returns the squirrel language definition.

func SshConfigLanguage 

func SshConfigLanguage() *gotreesitter.Language

SshConfigLanguage returns the ssh_config language definition.

func StarlarkLanguage 

func StarlarkLanguage() *gotreesitter.Language

StarlarkLanguage returns the starlark language definition.

func SvelteLanguage 

func SvelteLanguage() *gotreesitter.Language

SvelteLanguage returns the svelte language definition.

func SwiftLanguage 

func SwiftLanguage() *gotreesitter.Language

SwiftLanguage returns the swift language definition.

func TablegenLanguage 

func TablegenLanguage() *gotreesitter.Language

TablegenLanguage returns the tablegen language definition.

func TclLanguage 

func TclLanguage() *gotreesitter.Language

TclLanguage returns the tcl language definition.

func TealLanguage 

func TealLanguage() *gotreesitter.Language

TealLanguage returns the teal language definition.

func TemplLanguage 

func TemplLanguage() *gotreesitter.Language

TemplLanguage returns the templ language definition.

func TextprotoLanguage 

func TextprotoLanguage() *gotreesitter.Language

TextprotoLanguage returns the textproto language definition.

func ThriftLanguage 

func ThriftLanguage() *gotreesitter.Language

ThriftLanguage returns the thrift language definition.

func TlaplusLanguage 

func TlaplusLanguage() *gotreesitter.Language

TlaplusLanguage returns the tlaplus language definition.

func TmuxLanguage 

func TmuxLanguage() *gotreesitter.Language

TmuxLanguage returns the tmux language definition.

func TodotxtLanguage 

func TodotxtLanguage() *gotreesitter.Language

TodotxtLanguage returns the todotxt language definition.

func TomlLanguage 

func TomlLanguage() *gotreesitter.Language

TomlLanguage returns the toml language definition.

func TsxLanguage 

func TsxLanguage() *gotreesitter.Language

TsxLanguage returns the tsx language definition.

func TurtleLanguage 

func TurtleLanguage() *gotreesitter.Language

TurtleLanguage returns the turtle language definition.

func TwigLanguage 

func TwigLanguage() *gotreesitter.Language

TwigLanguage returns the twig language definition.

func TypescriptLanguage 

func TypescriptLanguage() *gotreesitter.Language

TypescriptLanguage returns the typescript language definition.

func TypstLanguage 

func TypstLanguage() *gotreesitter.Language

TypstLanguage returns the typst language definition.

func UnloadEmbeddedLanguage 

func UnloadEmbeddedLanguage(blobName string) bool

UnloadEmbeddedLanguage removes one grammar blob from the decoded cache. Existing parser instances that already reference the language remain valid.

func UxntalLanguage 

func UxntalLanguage() *gotreesitter.Language

UxntalLanguage returns the uxntal language definition.

func VLanguage 

func VLanguage() *gotreesitter.Language

VLanguage returns the v language definition.

func VerilogLanguage 

func VerilogLanguage() *gotreesitter.Language

VerilogLanguage returns the verilog language definition.

func VhdlLanguage 

func VhdlLanguage() *gotreesitter.Language

VhdlLanguage returns the vhdl language definition.

func VimdocLanguage 

func VimdocLanguage() *gotreesitter.Language

VimdocLanguage returns the vimdoc language definition.

func VueLanguage 

func VueLanguage() *gotreesitter.Language

VueLanguage returns the vue language definition.

func WalkAndParse 

func WalkAndParse(ctx context.Context, root string, policy ParsePolicy) (<-chan ParsedFile, func() WalkStats)

WalkAndParse walks root, discovers source files, and streams parsed results on the returned channel. The caller must drain the channel to completion. The returned function provides aggregate statistics and blocks until the walk is fully complete.

Pipeline:

  1. Walk with filepath.WalkDir, skipping SkipDirs, ShouldSkipDir, and SkipExtensions.
  2. Detect language via DetectLanguage; skip unknown files.
  3. Skip binary files (NUL byte in first 8 KB).
  4. Call ShouldParse hook if set; skip if it returns false.
  5. Normal files (< LargeFileThreshold): acquire 1 semaphore slot, read+parse in a goroutine, send result, then release.
  6. Large files (>= LargeFileThreshold): acquire ALL slots, parse inline, send result, release ALL slots.

Backpressure: workers release the semaphore AFTER sending on the channel, not before. ChannelBuffer must be at least MaxConcurrent+1 to prevent deadlock. A slow consumer naturally throttles the producer because workers block on the channel send while holding their semaphore slot.

Cancellation: pass a cancellable context to stop the walk early. In-flight parses may complete, but no new files will be dispatched. The channel will close promptly after cancellation.

Usage: 

policy := grammars.DefaultPolicy()
ch, statsFn := grammars.WalkAndParse(ctx, "/path/to/repo", policy)

for pf := range ch {
    if pf.Err != nil {
        log.Printf("error: %s: %v", pf.Path, pf.Err)
        pf.Close()
        continue
    }
    root := pf.Tree.RootNode()
    // ... inspect the AST ...
    pf.Close() // MUST call to release tree memory
}

stats := statsFn() // blocks until fully done; safe to call after draining
fmt.Printf("parsed %d files (%d bytes)\n", stats.FilesParsed, stats.BytesParsed)

func WatLanguage 

func WatLanguage() *gotreesitter.Language

WatLanguage returns the wat language definition.

func WgslLanguage 

func WgslLanguage() *gotreesitter.Language

WgslLanguage returns the wgsl language definition.

func WolframLanguage 

func WolframLanguage() *gotreesitter.Language

WolframLanguage returns the wolfram language definition.

func XmlLanguage 

func XmlLanguage() *gotreesitter.Language

XmlLanguage returns the xml language definition.

func YamlLanguage 

func YamlLanguage() *gotreesitter.Language

YamlLanguage returns the yaml language definition.

func YuckLanguage 

func YuckLanguage() *gotreesitter.Language

YuckLanguage returns the yuck language definition.

func ZigLanguage 

func ZigLanguage() *gotreesitter.Language

ZigLanguage returns the zig language definition.

Types

type AgdaExternalScanner 

type AgdaExternalScanner struct{}

AgdaExternalScanner handles newline/indent/dedent for Agda.

func (AgdaExternalScanner) Create 

func (AgdaExternalScanner) Create() any

func (AgdaExternalScanner) Deserialize 

func (AgdaExternalScanner) Deserialize(payload any, buf []byte)

func (AgdaExternalScanner) Destroy 

func (AgdaExternalScanner) Destroy(payload any)

func (AgdaExternalScanner) Scan 

func (AgdaExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (AgdaExternalScanner) Serialize 

func (AgdaExternalScanner) Serialize(payload any, buf []byte) int

type AngularExternalScanner 

type AngularExternalScanner struct{}

AngularExternalScanner handles HTML tag tracking plus Angular-specific interpolation for Angular templates.

func (AngularExternalScanner) Create 

func (AngularExternalScanner) Create() any

func (AngularExternalScanner) Deserialize 

func (AngularExternalScanner) Deserialize(payload any, buf []byte)

func (AngularExternalScanner) Destroy 

func (AngularExternalScanner) Destroy(payload any)

func (AngularExternalScanner) Scan 

func (AngularExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (AngularExternalScanner) Serialize 

func (AngularExternalScanner) Serialize(payload any, buf []byte) int

type ArduinoExternalScanner 

type ArduinoExternalScanner struct{}

ArduinoExternalScanner handles C++ R"delim(...)delim" raw string literals for Arduino.

func (ArduinoExternalScanner) Create 

func (ArduinoExternalScanner) Create() any

func (ArduinoExternalScanner) Deserialize 

func (ArduinoExternalScanner) Deserialize(payload any, buf []byte)

func (ArduinoExternalScanner) Destroy 

func (ArduinoExternalScanner) Destroy(payload any)

func (ArduinoExternalScanner) Scan 

func (ArduinoExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ArduinoExternalScanner) Serialize 

func (ArduinoExternalScanner) Serialize(payload any, buf []byte) int

type AstroExternalScanner 

type AstroExternalScanner struct{}

AstroExternalScanner handles Astro-specific external scanning: HTML tag tracking, fragment tags, frontmatter, JS expressions, backtick strings, interpolation, and permissible text.

func (AstroExternalScanner) Create 

func (AstroExternalScanner) Create() any

func (AstroExternalScanner) Deserialize 

func (AstroExternalScanner) Deserialize(payload any, buf []byte)

func (AstroExternalScanner) Destroy 

func (AstroExternalScanner) Destroy(any)

func (AstroExternalScanner) Scan 

func (AstroExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (AstroExternalScanner) Serialize 

func (AstroExternalScanner) Serialize(payload any, buf []byte) int

type AuthzedTokenSource 

type AuthzedTokenSource struct {
	// contains filtered or unexported fields
}

AuthzedTokenSource is a custom lexer for the authzed (SpiceDB/Zanzibar) permission language. The grammar is derived from tree-sitter-go and has keyword-like named tokens (definition_literal, relation_literal, etc.) that the generic lexer cannot handle, plus significant newlines.

func NewAuthzedTokenSource 

func NewAuthzedTokenSource(src []byte, lang *gotreesitter.Language) (*AuthzedTokenSource, error)

NewAuthzedTokenSource creates a token source for authzed source text.

func (*AuthzedTokenSource) Next 

func (ts *AuthzedTokenSource) Next() gotreesitter.Token

Next returns the next token from the source.

func (*AuthzedTokenSource) Reset 

func (ts *AuthzedTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*AuthzedTokenSource) SetGLRStates 

func (ts *AuthzedTokenSource) SetGLRStates(states []gotreesitter.StateID)

func (*AuthzedTokenSource) SetParserState 

func (ts *AuthzedTokenSource) SetParserState(state gotreesitter.StateID)

func (*AuthzedTokenSource) SkipToByte 

func (ts *AuthzedTokenSource) SkipToByte(offset uint32) gotreesitter.Token

SkipToByte advances to the given byte offset and returns the next token.

func (*AuthzedTokenSource) SupportsIncrementalReuse 

func (ts *AuthzedTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that AuthzedTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type AwkExternalScanner 

type AwkExternalScanner struct{}

AwkExternalScanner handles spacing-sensitive tokens for AWK.

func (AwkExternalScanner) Create 

func (AwkExternalScanner) Create() any

func (AwkExternalScanner) Deserialize 

func (AwkExternalScanner) Deserialize(payload any, buf []byte)

func (AwkExternalScanner) Destroy 

func (AwkExternalScanner) Destroy(payload any)

func (AwkExternalScanner) Scan 

func (AwkExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (AwkExternalScanner) Serialize 

func (AwkExternalScanner) Serialize(payload any, buf []byte) int

type BashExternalScanner 

type BashExternalScanner struct{}

BashExternalScanner implements gotreesitter.ExternalScanner for the Bash grammar.

func (BashExternalScanner) Create 

func (BashExternalScanner) Create() any

func (BashExternalScanner) Deserialize 

func (BashExternalScanner) Deserialize(payload any, buf []byte)

func (BashExternalScanner) Destroy 

func (BashExternalScanner) Destroy(payload any)

func (BashExternalScanner) Scan 

func (BashExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (BashExternalScanner) Serialize 

func (BashExternalScanner) Serialize(payload any, buf []byte) int

type BeancountExternalScanner 

type BeancountExternalScanner struct{}

BeancountExternalScanner handles org-mode style section headers and EOF for Beancount.

func (BeancountExternalScanner) Create 

func (BeancountExternalScanner) Create() any

func (BeancountExternalScanner) Deserialize 

func (BeancountExternalScanner) Deserialize(payload any, buf []byte)

func (BeancountExternalScanner) Destroy 

func (BeancountExternalScanner) Destroy(payload any)

func (BeancountExternalScanner) Scan 

func (BeancountExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (BeancountExternalScanner) Serialize 

func (BeancountExternalScanner) Serialize(payload any, buf []byte) int

type BicepExternalScanner 

type BicepExternalScanner struct{}

BicepExternalScanner handles wildcard resource type matching and triple-single-quote multiline strings for Bicep.

func (BicepExternalScanner) Create 

func (BicepExternalScanner) Create() any

func (BicepExternalScanner) Deserialize 

func (BicepExternalScanner) Deserialize(payload any, buf []byte)

func (BicepExternalScanner) Destroy 

func (BicepExternalScanner) Destroy(payload any)

func (BicepExternalScanner) Scan 

func (BicepExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (BicepExternalScanner) Serialize 

func (BicepExternalScanner) Serialize(payload any, buf []byte) int

type BitbakeExternalScanner 

type BitbakeExternalScanner struct{}

BitbakeExternalScanner handles Python-like indent/dedent, strings, concat, and shell content.

func (BitbakeExternalScanner) Create 

func (BitbakeExternalScanner) Create() any

func (BitbakeExternalScanner) Deserialize 

func (BitbakeExternalScanner) Deserialize(payload any, buf []byte)

func (BitbakeExternalScanner) Destroy 

func (BitbakeExternalScanner) Destroy(payload any)

func (BitbakeExternalScanner) Scan 

func (BitbakeExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (BitbakeExternalScanner) Serialize 

func (BitbakeExternalScanner) Serialize(payload any, buf []byte) int

type BladeExternalScanner 

type BladeExternalScanner struct{}

BladeExternalScanner handles HTML tag tracking for Blade templates.

func (BladeExternalScanner) Create 

func (BladeExternalScanner) Create() any

func (BladeExternalScanner) Deserialize 

func (BladeExternalScanner) Deserialize(payload any, buf []byte)

func (BladeExternalScanner) Destroy 

func (BladeExternalScanner) Destroy(payload any)

func (BladeExternalScanner) Scan 

func (BladeExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (BladeExternalScanner) Serialize 

func (BladeExternalScanner) Serialize(payload any, buf []byte) int

type CSharpExternalScanner 

type CSharpExternalScanner struct{}

CSharpExternalScanner handles auto-semicolons, interpolated strings, and raw strings for C#.

func (CSharpExternalScanner) Create 

func (CSharpExternalScanner) Create() any

func (CSharpExternalScanner) Deserialize 

func (CSharpExternalScanner) Deserialize(payload any, buf []byte)

func (CSharpExternalScanner) Destroy 

func (CSharpExternalScanner) Destroy(payload any)

func (CSharpExternalScanner) Scan 

func (CSharpExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CSharpExternalScanner) Serialize 

func (CSharpExternalScanner) Serialize(payload any, buf []byte) int

type CTokenSource 

type CTokenSource struct {
	// contains filtered or unexported fields
}

CTokenSource is a lightweight lexer bridge for tree-sitter-c.

func NewCTokenSource 

func NewCTokenSource(src []byte, lang *gotreesitter.Language) (*CTokenSource, error)

NewCTokenSource creates a token source for C source text.

func (*CTokenSource) Close 

func (ts *CTokenSource) Close()

Close clears parser-owned state and returns this token source to the pool.

func (*CTokenSource) Next 

func (ts *CTokenSource) Next() gotreesitter.Token

func (*CTokenSource) RebuildTokenSource 

func (ts *CTokenSource) RebuildTokenSource(src []byte, lang *gotreesitter.Language) (gotreesitter.TokenSource, error)

RebuildTokenSource constructs a fresh C token source for another source buffer while preserving the grammar table identity.

func (*CTokenSource) Reset 

func (ts *CTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*CTokenSource) SetGLRStates 

func (ts *CTokenSource) SetGLRStates(states []gotreesitter.StateID)

func (*CTokenSource) SetParserState 

func (ts *CTokenSource) SetParserState(state gotreesitter.StateID)

func (*CTokenSource) SkipToByte 

func (ts *CTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*CTokenSource) SupportsIncrementalReuse 

func (ts *CTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that CTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type CaddyExternalScanner 

type CaddyExternalScanner struct{}

CaddyExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-caddy. Handles _newline, _indent, and _dedent tokens for indentation tracking.

func (CaddyExternalScanner) Create 

func (CaddyExternalScanner) Create() any

func (CaddyExternalScanner) Deserialize 

func (CaddyExternalScanner) Deserialize(payload any, buf []byte)

func (CaddyExternalScanner) Destroy 

func (CaddyExternalScanner) Destroy(payload any)

func (CaddyExternalScanner) Scan 

func (CaddyExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CaddyExternalScanner) Serialize 

func (CaddyExternalScanner) Serialize(payload any, buf []byte) int

type CairoExternalScanner 

type CairoExternalScanner struct{}

CairoExternalScanner handles %{ %} hint blocks with embedded Python in Cairo.

func (CairoExternalScanner) Create 

func (CairoExternalScanner) Create() any

func (CairoExternalScanner) Deserialize 

func (CairoExternalScanner) Deserialize(payload any, buf []byte)

func (CairoExternalScanner) Destroy 

func (CairoExternalScanner) Destroy(payload any)

func (CairoExternalScanner) Scan 

func (CairoExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CairoExternalScanner) Serialize 

func (CairoExternalScanner) Serialize(payload any, buf []byte) int

type CmakeExternalScanner 

type CmakeExternalScanner struct{}

CmakeExternalScanner handles CMake bracket arguments, bracket comments, and line comments.

func (CmakeExternalScanner) Create 

func (CmakeExternalScanner) Create() any

func (CmakeExternalScanner) Deserialize 

func (CmakeExternalScanner) Deserialize(payload any, buf []byte)

func (CmakeExternalScanner) Destroy 

func (CmakeExternalScanner) Destroy(payload any)

func (CmakeExternalScanner) PreservesStateOnScanFailure 

func (CmakeExternalScanner) PreservesStateOnScanFailure() bool

func (CmakeExternalScanner) Scan 

func (CmakeExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CmakeExternalScanner) Serialize 

func (CmakeExternalScanner) Serialize(payload any, buf []byte) int

func (CmakeExternalScanner) SupportsIncrementalReuse 

func (CmakeExternalScanner) SupportsIncrementalReuse() bool

type CobolExternalScanner 

type CobolExternalScanner struct{}

CobolExternalScanner handles COBOL's column-based formatting.

func (CobolExternalScanner) Create 

func (CobolExternalScanner) Create() any

func (CobolExternalScanner) Deserialize 

func (CobolExternalScanner) Deserialize(payload any, buf []byte)

func (CobolExternalScanner) Destroy 

func (CobolExternalScanner) Destroy(payload any)

func (CobolExternalScanner) Scan 

func (CobolExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CobolExternalScanner) Serialize 

func (CobolExternalScanner) Serialize(payload any, buf []byte) int

type CommentExternalScanner 

type CommentExternalScanner struct{}

CommentExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-comment.

The comment grammar (tree-sitter-comment) parses structured comment text such as "TODO: fix this" or "FIXME(user): description". The external scanner is responsible for producing the "name" token which represents a tag keyword (TODO, FIXME, NOTE, HACK, etc.).

The scanner must be careful not to match arbitrary text as a "name", since the DFA handles regular text via _text_token1. A name is only returned when the scanned word is immediately followed by ':' or '(', indicating it forms part of a tag construct.

func (CommentExternalScanner) Create 

func (CommentExternalScanner) Create() any

func (CommentExternalScanner) Deserialize 

func (CommentExternalScanner) Deserialize(payload any, buf []byte)

func (CommentExternalScanner) Destroy 

func (CommentExternalScanner) Destroy(payload any)

func (CommentExternalScanner) Scan 

func (CommentExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CommentExternalScanner) Serialize 

func (CommentExternalScanner) Serialize(payload any, buf []byte) int

type CooklangExternalScanner 

type CooklangExternalScanner struct{}

CooklangExternalScanner handles newline detection for Cooklang recipe files.

func (CooklangExternalScanner) Create 

func (CooklangExternalScanner) Create() any

func (CooklangExternalScanner) Deserialize 

func (CooklangExternalScanner) Deserialize(payload any, buf []byte)

func (CooklangExternalScanner) Destroy 

func (CooklangExternalScanner) Destroy(payload any)

func (CooklangExternalScanner) Scan 

func (CooklangExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CooklangExternalScanner) Serialize 

func (CooklangExternalScanner) Serialize(payload any, buf []byte) int

type CppExternalScanner 

type CppExternalScanner struct{}

CppExternalScanner handles C++ R"delim(...)delim" raw string literals.

func (CppExternalScanner) Create 

func (CppExternalScanner) Create() any

func (CppExternalScanner) Deserialize 

func (CppExternalScanner) Deserialize(payload any, buf []byte)

func (CppExternalScanner) Destroy 

func (CppExternalScanner) Destroy(payload any)

func (CppExternalScanner) Scan 

func (CppExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CppExternalScanner) Serialize 

func (CppExternalScanner) Serialize(payload any, buf []byte) int

type CrystalExternalScanner 

type CrystalExternalScanner struct{}

func (CrystalExternalScanner) Create 

func (CrystalExternalScanner) Create() any

func (CrystalExternalScanner) Deserialize 

func (CrystalExternalScanner) Deserialize(payload any, buf []byte)

func (CrystalExternalScanner) Destroy 

func (CrystalExternalScanner) Destroy(payload any)

func (CrystalExternalScanner) Scan 

func (CrystalExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CrystalExternalScanner) Serialize 

func (CrystalExternalScanner) Serialize(payload any, buf []byte) int

type CssExternalScanner 

type CssExternalScanner struct{}

CssExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-css.

This is a Go port of the C external scanner from tree-sitter-css. The scanner handles three tokens:

  • _descendant_operator: whitespace between two selectors (descendant combinator)
  • pseudo_class_selector_colon: a ":" that starts a pseudo-class (vs property-value separator)
  • __error_recovery: sentinel that causes immediate bail-out

The key challenge is contextual disambiguation: whitespace might be a descendant combinator or just formatting, and ":" might start a pseudo-class or separate a property from its value.

func (CssExternalScanner) Create 

func (CssExternalScanner) Create() any

func (CssExternalScanner) Deserialize 

func (CssExternalScanner) Deserialize(payload any, buf []byte)

func (CssExternalScanner) Destroy 

func (CssExternalScanner) Destroy(payload any)

func (CssExternalScanner) Scan 

func (CssExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CssExternalScanner) Serialize 

func (CssExternalScanner) Serialize(payload any, buf []byte) int

func (CssExternalScanner) SupportsIncrementalReuse 

func (CssExternalScanner) SupportsIncrementalReuse() bool

type CudaExternalScanner 

type CudaExternalScanner struct{}

CudaExternalScanner handles C++ R"delim(...)delim" raw string literals for CUDA.

func (CudaExternalScanner) Create 

func (CudaExternalScanner) Create() any

func (CudaExternalScanner) Deserialize 

func (CudaExternalScanner) Deserialize(payload any, buf []byte)

func (CudaExternalScanner) Destroy 

func (CudaExternalScanner) Destroy(payload any)

func (CudaExternalScanner) Scan 

func (CudaExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CudaExternalScanner) Serialize 

func (CudaExternalScanner) Serialize(payload any, buf []byte) int

type CueExternalScanner 

type CueExternalScanner struct{}

CueExternalScanner handles string content scanning for CUE's various string types: multi-line, raw, and multi-line raw strings/bytes.

func (CueExternalScanner) Create 

func (CueExternalScanner) Create() any

func (CueExternalScanner) Deserialize 

func (CueExternalScanner) Deserialize(payload any, buf []byte)

func (CueExternalScanner) Destroy 

func (CueExternalScanner) Destroy(payload any)

func (CueExternalScanner) Scan 

func (CueExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (CueExternalScanner) Serialize 

func (CueExternalScanner) Serialize(payload any, buf []byte) int

type DExternalScanner 

type DExternalScanner struct{}

DExternalScanner handles external tokens for the D grammar. Ported from tree-sitter-d/src/scanner.c.

func (DExternalScanner) Create 

func (DExternalScanner) Create() any

func (DExternalScanner) Deserialize 

func (DExternalScanner) Deserialize(payload any, buf []byte)

func (DExternalScanner) Destroy 

func (DExternalScanner) Destroy(payload any)

func (DExternalScanner) Scan 

func (DExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DExternalScanner) Serialize 

func (DExternalScanner) Serialize(payload any, buf []byte) int

type DartExternalScanner 

type DartExternalScanner struct {
	// contains filtered or unexported fields
}

DartExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-dart.

This is a Go port of the C external scanner from UserNobody14/tree-sitter-dart. The scanner is stateless and handles:

  • Template/string content tokens for 4 string variants (single/double x single/multi-line)
  • Raw string backslash passthrough
  • Nestable block comments (/* */ and /** */)

func (DartExternalScanner) Create 

func (DartExternalScanner) Create() any

func (DartExternalScanner) Deserialize 

func (DartExternalScanner) Deserialize(payload any, buf []byte)

func (DartExternalScanner) Destroy 

func (DartExternalScanner) Destroy(payload any)

func (DartExternalScanner) ExternalScannerForLanguage 

func (DartExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

func (DartExternalScanner) Scan 

func (s DartExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DartExternalScanner) Serialize 

func (DartExternalScanner) Serialize(payload any, buf []byte) int

func (DartExternalScanner) SupportsIncrementalReuse 

func (DartExternalScanner) SupportsIncrementalReuse() bool

type DhallExternalScanner 

type DhallExternalScanner struct{}

DhallExternalScanner handles nestable {- -} block comments for Dhall.

func (DhallExternalScanner) Create 

func (DhallExternalScanner) Create() any

func (DhallExternalScanner) Deserialize 

func (DhallExternalScanner) Deserialize(payload any, buf []byte)

func (DhallExternalScanner) Destroy 

func (DhallExternalScanner) Destroy(payload any)

func (DhallExternalScanner) Scan 

func (DhallExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DhallExternalScanner) Serialize 

func (DhallExternalScanner) Serialize(payload any, buf []byte) int

type DisassemblyExternalScanner 

type DisassemblyExternalScanner struct{}

DisassemblyExternalScanner handles assembly instruction vs memory dump disambiguation.

func (DisassemblyExternalScanner) Create 

func (DisassemblyExternalScanner) Create() any

func (DisassemblyExternalScanner) Deserialize 

func (DisassemblyExternalScanner) Deserialize(payload any, buf []byte)

func (DisassemblyExternalScanner) Destroy 

func (DisassemblyExternalScanner) Destroy(payload any)

func (DisassemblyExternalScanner) Scan 

func (DisassemblyExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DisassemblyExternalScanner) Serialize 

func (DisassemblyExternalScanner) Serialize(payload any, buf []byte) int

type DjotExternalScanner 

type DjotExternalScanner struct{}

DjotExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-djot.

func (DjotExternalScanner) Create 

func (DjotExternalScanner) Create() any

func (DjotExternalScanner) Deserialize 

func (DjotExternalScanner) Deserialize(payload any, buf []byte)

func (DjotExternalScanner) Destroy 

func (DjotExternalScanner) Destroy(payload any)

func (DjotExternalScanner) Scan 

func (DjotExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DjotExternalScanner) Serialize 

func (DjotExternalScanner) Serialize(payload any, buf []byte) int

type DockerfileExternalScanner 

type DockerfileExternalScanner struct{}

DockerfileExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-dockerfile.

This is a Go port of the C external scanner from camdencheek/tree-sitter-dockerfile. The scanner manages Dockerfile heredoc syntax (<<MARKER / <<-MARKER) with a stack of up to 10 delimiter strings and handles:

  • heredoc_marker: the <<[-]DELIM opening
  • heredoc_line: content lines within a heredoc
  • heredoc_end: the closing delimiter line
  • _heredoc_nl: newlines within heredoc context
  • error_sentinel: error recovery bail-out

func (DockerfileExternalScanner) Create 

func (DockerfileExternalScanner) Create() any

func (DockerfileExternalScanner) Deserialize 

func (DockerfileExternalScanner) Deserialize(payload any, buf []byte)

func (DockerfileExternalScanner) Destroy 

func (DockerfileExternalScanner) Destroy(payload any)

func (DockerfileExternalScanner) Scan 

func (DockerfileExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DockerfileExternalScanner) Serialize 

func (DockerfileExternalScanner) Serialize(payload any, buf []byte) int

type DoxygenExternalScanner 

type DoxygenExternalScanner struct{}

DoxygenExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-doxygen. The doxygen grammar parses documentation comment body text (without the // or /* comment markers).

Five external tokens are handled:

  • brief_text: captures text after @brief/@short/\brief/\short until EOL
  • code_block_start: matches @code or \code
  • code_block_language: matches {.lang} immediately after code_block_start
  • code_block_content: scans all text until @endcode or \endcode
  • code_block_end: matches @endcode or \endcode

func (DoxygenExternalScanner) Create 

func (DoxygenExternalScanner) Create() any

func (DoxygenExternalScanner) Deserialize 

func (DoxygenExternalScanner) Deserialize(payload any, buf []byte)

func (DoxygenExternalScanner) Destroy 

func (DoxygenExternalScanner) Destroy(payload any)

func (DoxygenExternalScanner) Scan 

func (DoxygenExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DoxygenExternalScanner) Serialize 

func (DoxygenExternalScanner) Serialize(payload any, buf []byte) int

type DtdExternalScanner 

type DtdExternalScanner struct{}

DtdExternalScanner handles processing instructions and <!-- --> comments for DTD.

func (DtdExternalScanner) Create 

func (DtdExternalScanner) Create() any

func (DtdExternalScanner) Deserialize 

func (DtdExternalScanner) Deserialize(payload any, buf []byte)

func (DtdExternalScanner) Destroy 

func (DtdExternalScanner) Destroy(payload any)

func (DtdExternalScanner) Scan 

func (DtdExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (DtdExternalScanner) Serialize 

func (DtdExternalScanner) Serialize(payload any, buf []byte) int

type EarthfileExternalScanner 

type EarthfileExternalScanner struct{}

EarthfileExternalScanner handles indent/dedent for Earthfile.

func (EarthfileExternalScanner) Create 

func (EarthfileExternalScanner) Create() any

func (EarthfileExternalScanner) Deserialize 

func (EarthfileExternalScanner) Deserialize(payload any, buf []byte)

func (EarthfileExternalScanner) Destroy 

func (EarthfileExternalScanner) Destroy(payload any)

func (EarthfileExternalScanner) Scan 

func (EarthfileExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (EarthfileExternalScanner) Serialize 

func (EarthfileExternalScanner) Serialize(payload any, buf []byte) int

type EditorconfigExternalScanner 

type EditorconfigExternalScanner struct{}

EditorconfigExternalScanner handles EOF and integer-range detection for .editorconfig files.

func (EditorconfigExternalScanner) Create 

func (EditorconfigExternalScanner) Create() any

func (EditorconfigExternalScanner) Deserialize 

func (EditorconfigExternalScanner) Deserialize(payload any, buf []byte)

func (EditorconfigExternalScanner) Destroy 

func (EditorconfigExternalScanner) Destroy(payload any)

func (EditorconfigExternalScanner) Scan 

func (EditorconfigExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (EditorconfigExternalScanner) Serialize 

func (EditorconfigExternalScanner) Serialize(payload any, buf []byte) int

type ElixirExternalScanner 

type ElixirExternalScanner struct{}

func (ElixirExternalScanner) Create 

func (ElixirExternalScanner) Create() any

func (ElixirExternalScanner) Deserialize 

func (ElixirExternalScanner) Deserialize(payload any, buf []byte)

func (ElixirExternalScanner) Destroy 

func (ElixirExternalScanner) Destroy(payload any)

func (ElixirExternalScanner) Scan 

func (ElixirExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ElixirExternalScanner) Serialize 

func (ElixirExternalScanner) Serialize(payload any, buf []byte) int

type ElmExternalScanner 

type ElmExternalScanner struct{}

ElmExternalScanner handles indentation-based layout for Elm.

func (ElmExternalScanner) Create 

func (ElmExternalScanner) Create() any

func (ElmExternalScanner) Deserialize 

func (ElmExternalScanner) Deserialize(payload any, buf []byte)

func (ElmExternalScanner) Destroy 

func (ElmExternalScanner) Destroy(payload any)

func (ElmExternalScanner) Scan 

func (ElmExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ElmExternalScanner) Serialize 

func (ElmExternalScanner) Serialize(payload any, buf []byte) int

type ErlangExternalScanner 

type ErlangExternalScanner struct{}

ErlangExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-erlang.

This is a Go port of the C external scanner from tree-sitter-erlang (WhatsApp/tree-sitter-erlang). The scanner handles Erlang's triple-quoted strings (EEP-0064): strings delimited by 3+ quote characters, where the closing delimiter must appear at the start of a line (after optional whitespace) and match the same number of quotes. Sigil strings optionally have a ~[sSbB]? prefix.

func (ErlangExternalScanner) Create 

func (ErlangExternalScanner) Create() any

func (ErlangExternalScanner) Deserialize 

func (ErlangExternalScanner) Deserialize(payload any, buf []byte)

func (ErlangExternalScanner) Destroy 

func (ErlangExternalScanner) Destroy(payload any)

func (ErlangExternalScanner) Scan 

func (ErlangExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ErlangExternalScanner) Serialize 

func (ErlangExternalScanner) Serialize(payload any, buf []byte) int

type ExtensionEntry 

type ExtensionEntry struct {
	Name              string
	Extensions        []string // file extensions: [".dmj", ".dingo", ".fw"]
	Aliases           []string // markdown fence aliases: ["dmj", "danmuji"]
	GenerateLanguage  func() (*gotreesitter.Language, error)
	HighlightQuery    string
	InheritHighlights string // parent language for highlight query composition (e.g. "go")
}

RegisterExtension registers a grammargen-based grammar extension with the language registry. This enables detection by file extension, markdown code fence highlighting, and LSP support. The language is generated lazily on first access. Its result, including a generation error, is cached so the generator is invoked at most once.

Usage from an extension package: 

func init() {
    grammars.RegisterExtension(grammars.ExtensionEntry{
        Name:           "danmuji",
        Extensions:     []string{".dmj"},
        Aliases:        []string{"dmj"},
        GenerateLanguage: func() (*gotreesitter.Language, error) {
            return grammargen.GenerateLanguage(danmuji.Grammar())
        },
        HighlightQuery: danmuji.HighlightQueries(),
    })
}

type ExternalScannerSourceFile 

type ExternalScannerSourceFile struct {
	Path   string
	SHA256 string
}

ExternalScannerSourceFile identifies an upstream scanner-facing source file. The hash is SHA-256 over the exact file bytes at ExternalScannerSpec.UpstreamCommit.

type ExternalScannerSpec 

type ExternalScannerSpec struct {
	Language       string
	UpstreamRepo   string
	UpstreamCommit string
	SourceFiles    []ExternalScannerSourceFile
	Externals      []string
}

ExternalScannerSpec records the source contract for a hand-written external scanner port. It lets updater tooling distinguish grammar-only changes from external token or scanner-source changes that require scanner review.

func ExternalScannerSpecs 

func ExternalScannerSpecs() []ExternalScannerSpec

ExternalScannerSpecs returns all registered scanner-source specs, sorted by language.

func LookupExternalScannerSpec 

func LookupExternalScannerSpec(name string) (ExternalScannerSpec, bool)

LookupExternalScannerSpec returns scanner-source metadata for a language.

type FennelExternalScanner 

type FennelExternalScanner struct{}

FennelExternalScanner handles reader macros and shebang for Fennel.

func (FennelExternalScanner) Create 

func (FennelExternalScanner) Create() any

func (FennelExternalScanner) Deserialize 

func (FennelExternalScanner) Deserialize(payload any, buf []byte)

func (FennelExternalScanner) Destroy 

func (FennelExternalScanner) Destroy(payload any)

func (FennelExternalScanner) Scan 

func (FennelExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FennelExternalScanner) Serialize 

func (FennelExternalScanner) Serialize(payload any, buf []byte) int

type FirrtlExternalScanner 

type FirrtlExternalScanner struct{}

FirrtlExternalScanner handles newline/indent/dedent for FIRRTL.

func (FirrtlExternalScanner) Create 

func (FirrtlExternalScanner) Create() any

func (FirrtlExternalScanner) Deserialize 

func (FirrtlExternalScanner) Deserialize(payload any, buf []byte)

func (FirrtlExternalScanner) Destroy 

func (FirrtlExternalScanner) Destroy(payload any)

func (FirrtlExternalScanner) Scan 

func (FirrtlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FirrtlExternalScanner) Serialize 

func (FirrtlExternalScanner) Serialize(payload any, buf []byte) int

type FishExternalScanner 

type FishExternalScanner struct{}

FishExternalScanner handles concatenation detection and begin-brace disambiguation for the Fish shell.

func (FishExternalScanner) Create 

func (FishExternalScanner) Create() any

func (FishExternalScanner) Deserialize 

func (FishExternalScanner) Deserialize(payload any, buf []byte)

func (FishExternalScanner) Destroy 

func (FishExternalScanner) Destroy(payload any)

func (FishExternalScanner) Scan 

func (FishExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FishExternalScanner) Serialize 

func (FishExternalScanner) Serialize(payload any, buf []byte) int

type FoamExternalScanner 

type FoamExternalScanner struct{}

FoamExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-foam.

This is a Go port of the C external scanner from tree-sitter-foam (https://github.com/FoamScience/tree-sitter-foam). The scanner handles:

  • identifier: OpenFOAM identifiers (keyword names, paths, etc.)
  • boolean: "on", "off", "true", "false"
  • _eof: end-of-file marker

func (FoamExternalScanner) Create 

func (FoamExternalScanner) Create() any

func (FoamExternalScanner) Deserialize 

func (FoamExternalScanner) Deserialize(payload any, buf []byte)

func (FoamExternalScanner) Destroy 

func (FoamExternalScanner) Destroy(payload any)

func (FoamExternalScanner) Scan 

func (FoamExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FoamExternalScanner) Serialize 

func (FoamExternalScanner) Serialize(payload any, buf []byte) int

type FortranExternalScanner 

type FortranExternalScanner struct{}

FortranExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-fortran.

func (FortranExternalScanner) Create 

func (FortranExternalScanner) Create() any

func (FortranExternalScanner) Deserialize 

func (FortranExternalScanner) Deserialize(payload any, buf []byte)

func (FortranExternalScanner) Destroy 

func (FortranExternalScanner) Destroy(payload any)

func (FortranExternalScanner) Scan 

func (FortranExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FortranExternalScanner) Serialize 

func (FortranExternalScanner) Serialize(payload any, buf []byte) int

type FsharpExternalScanner 

type FsharpExternalScanner struct{}

FsharpExternalScanner handles indent/dedent, keywords, preprocessor directives, and comments for F#.

func (FsharpExternalScanner) Create 

func (FsharpExternalScanner) Create() any

func (FsharpExternalScanner) Deserialize 

func (FsharpExternalScanner) Deserialize(payload any, buf []byte)

func (FsharpExternalScanner) Destroy 

func (FsharpExternalScanner) Destroy(payload any)

func (FsharpExternalScanner) Scan 

func (FsharpExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (FsharpExternalScanner) Serialize 

func (FsharpExternalScanner) Serialize(payload any, buf []byte) int

type GdscriptExternalScanner 

type GdscriptExternalScanner struct{}

GdscriptExternalScanner handles indent/dedent, strings, and body_end for GDScript.

func (GdscriptExternalScanner) Create 

func (GdscriptExternalScanner) Create() any

func (GdscriptExternalScanner) Deserialize 

func (GdscriptExternalScanner) Deserialize(payload any, buf []byte)

func (GdscriptExternalScanner) Destroy 

func (GdscriptExternalScanner) Destroy(payload any)

func (GdscriptExternalScanner) Scan 

func (GdscriptExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (GdscriptExternalScanner) Serialize 

func (GdscriptExternalScanner) Serialize(payload any, buf []byte) int

type GenericTokenSource 

type GenericTokenSource struct {
	// contains filtered or unexported fields
}

GenericTokenSource is a best-effort scanner that maps source text to tree-sitter token symbols using token-name heuristics.

It is intended as a rollout bridge for grammars without DFA tables.

func NewGenericTokenSource 

func NewGenericTokenSource(src []byte, lang *gotreesitter.Language) (*GenericTokenSource, error)

NewGenericTokenSource creates a best-effort generic token source.

func (*GenericTokenSource) Next 

func (ts *GenericTokenSource) Next() gotreesitter.Token

func (*GenericTokenSource) Reset 

func (ts *GenericTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*GenericTokenSource) SkipToByte 

func (ts *GenericTokenSource) SkipToByte(offset uint32) gotreesitter.Token

type GitcommitExternalScanner 

type GitcommitExternalScanner struct{}

GitcommitExternalScanner handles conventional commit prefix detection.

func (GitcommitExternalScanner) Create 

func (GitcommitExternalScanner) Create() any

func (GitcommitExternalScanner) Deserialize 

func (GitcommitExternalScanner) Deserialize(payload any, buf []byte)

func (GitcommitExternalScanner) Destroy 

func (GitcommitExternalScanner) Destroy(payload any)

func (GitcommitExternalScanner) Scan 

func (GitcommitExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (GitcommitExternalScanner) Serialize 

func (GitcommitExternalScanner) Serialize(payload any, buf []byte) int

type GleamExternalScanner 

type GleamExternalScanner struct{}

GleamExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-gleam.

The gleam grammar uses an external scanner to produce two tokens:

  • quoted_content: the interior of a string literal, consuming characters until a closing " or escape \ is encountered.
  • doc_comment_content: a single line of a doc comment, consuming characters until end-of-line or EOF.

func (GleamExternalScanner) Create 

func (GleamExternalScanner) Create() any

func (GleamExternalScanner) Deserialize 

func (GleamExternalScanner) Deserialize(payload any, buf []byte)

func (GleamExternalScanner) Destroy 

func (GleamExternalScanner) Destroy(payload any)

func (GleamExternalScanner) Scan 

func (GleamExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (GleamExternalScanner) Serialize 

func (GleamExternalScanner) Serialize(payload any, buf []byte) int

type GnExternalScanner 

type GnExternalScanner struct{}

GnExternalScanner handles string content for GN (Generate Ninja) build files. Scans content inside "..." strings, stopping at closing quote, escape sequences, and ${...} interpolations.

func (GnExternalScanner) Create 

func (GnExternalScanner) Create() any

func (GnExternalScanner) Deserialize 

func (GnExternalScanner) Deserialize(payload any, buf []byte)

func (GnExternalScanner) Destroy 

func (GnExternalScanner) Destroy(payload any)

func (GnExternalScanner) Scan 

func (GnExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

Scan is a line-faithful port of the pinned upstream src/scanner.c (tree-sitter-grammars/tree-sitter-gn @ bc06955b).

func (GnExternalScanner) Serialize 

func (GnExternalScanner) Serialize(payload any, buf []byte) int

type GoExternalScanner 

type GoExternalScanner struct{}

GoExternalScanner resolves the Go grammar's `terminator` rule — automatic semicolon insertion (ASI) — via an external scanner instead of a plain DFA alternation.

Background: upstream tree-sitter-go has no scanner.c for this at all. Its grammar.js models ASI as `terminator = choice(/\n/, ';', '\0')`, and upstream's own generated parser gives every LR state its own lexer function, so the real newline pattern and the zero-width EOF sentinel never compete within a shared table. gotreesitter's grammargen backend instead compiles one shared, merged DFA across all states; at every terminator position the zero-width '\0' EOF sentinel and the one-byte `\n` pattern both accept, and the runtime's shared tie-break (parser_dfa_token_source.go: preferZeroWidthStartAcceptForState) prefers the zero-width accept unconditionally. That silently drops the trailing newline byte from the enclosing statement/declaration span everywhere except genuine end-of-file, which is the root cause of the Go C-parity gate failure this scanner fixes.

Routing `terminator`'s newline/EOF alternatives through a dedicated external scanner sidesteps the shared-DFA tie-break entirely: Scan is only ever invoked in parser states where a terminator is structurally legal (every occurrence in grammargen/go_grammar.go directly follows a completed statement, declaration, or spec, so there is no need to track "was the previous token an identifier/literal/keyword" — the grammar itself already gates that), and it makes an unambiguous decision from the raw byte stream with no shared-state tie-break involved. This mirrors how JavaScriptExternalScanner (grammars/javascript_scanner.go) resolves `_automatic_semicolon` for the JS/TS grammars in this package.

func (GoExternalScanner) Create 

func (GoExternalScanner) Create() any

func (GoExternalScanner) Deserialize 

func (GoExternalScanner) Deserialize(payload any, buf []byte)

func (GoExternalScanner) Destroy 

func (GoExternalScanner) Destroy(payload any)

func (GoExternalScanner) PreservesStateOnScanFailure 

func (GoExternalScanner) PreservesStateOnScanFailure() bool

PreservesStateOnScanFailure: Scan never mutates any persisted payload before returning false (there is no payload), so the token source can skip the snapshot/restore it would otherwise do around a failed scan.

func (GoExternalScanner) Scan 

func (GoExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (GoExternalScanner) Serialize 

func (GoExternalScanner) Serialize(payload any, buf []byte) int

func (GoExternalScanner) SupportsIncrementalReuse 

func (GoExternalScanner) SupportsIncrementalReuse() bool

SupportsIncrementalReuse: the scanner carries no serialized state (Create returns nil), so incremental subtree reuse is always safe.

type GoTokenSource 

type GoTokenSource struct {
	// contains filtered or unexported fields
}

GoTokenSource bridges Go's standard library scanner to tree-sitter's token format. It implements gotreesitter.TokenSource.

The tree-sitter Go grammar expects tokens at a finer granularity than go/scanner provides. In particular:

  • String literals are split into open-quote, content, close-quote
  • Raw string literals similarly split with backtick delimiters
  • Comments are emitted as explicit tokens
  • Newline-based automatic semicolons are mapped to ";"

func NewGoTokenSource 

func NewGoTokenSource(src []byte, lang *gotreesitter.Language) (*GoTokenSource, error)

NewGoTokenSource creates a token source that lexes Go source code and produces tree-sitter tokens compatible with the Go grammar.

func (*GoTokenSource) Next 

func (ts *GoTokenSource) Next() gotreesitter.Token

Next returns the next token. Returns a zero-Symbol token at EOF.

func (*GoTokenSource) RebuildTokenSource 

func (ts *GoTokenSource) RebuildTokenSource(src []byte, lang *gotreesitter.Language) (gotreesitter.TokenSource, error)

RebuildTokenSource constructs a fresh Go token source for the given source.

func (*GoTokenSource) Reset 

func (ts *GoTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*GoTokenSource) SkipToByte 

func (ts *GoTokenSource) SkipToByte(offset uint32) gotreesitter.Token

SkipToByte advances until it reaches the first token at or after offset.

func (*GoTokenSource) SkipToByteWithPoint 

func (ts *GoTokenSource) SkipToByteWithPoint(offset uint32, pt gotreesitter.Point) gotreesitter.Token

SkipToByteWithPoint jumps to offset using the provided Point, avoiding the O(n) offset-to-point scan that SkipToByte performs.

func (*GoTokenSource) SupportsIncrementalReuse 

func (ts *GoTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that GoTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte* behavior.

type GodotResourceExternalScanner 

type GodotResourceExternalScanner struct{}

GodotResourceExternalScanner handles multiline string literals in Godot .tres/.tscn resource files. Strings are "..." with \" escapes.

func (GodotResourceExternalScanner) Create 

func (GodotResourceExternalScanner) Create() any

func (GodotResourceExternalScanner) Deserialize 

func (GodotResourceExternalScanner) Deserialize(payload any, buf []byte)

func (GodotResourceExternalScanner) Destroy 

func (GodotResourceExternalScanner) Destroy(payload any)

func (GodotResourceExternalScanner) Scan 

func (GodotResourceExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

Scan is a line-faithful port of the pinned upstream src/scanner.c (PrestonKnopp/tree-sitter-godot-resource @ 302c1895).

Note the upstream escape handling: a quote terminates the string only when the PREVIOUS character was not a backslash. That means `\\"` does NOT terminate (the quote follows a backslash) — upstream does not treat `\\` as a completed escape pair. Parity requires reproducing that exactly.

func (GodotResourceExternalScanner) Serialize 

func (GodotResourceExternalScanner) Serialize(payload any, buf []byte) int

type GrammarSource 

type GrammarSource string

GrammarSource describes where a LangEntry's language loader comes from. 

const (
	GrammarSourceUnknown GrammarSource = "unknown"
	// GrammarSourceTS2GoBlob marks an embedded .bin blob compiled by the
	// ts2go pipeline from upstream C parser tables.
	GrammarSourceTS2GoBlob GrammarSource = "ts2go_blob"
	// GrammarSourceGrammargen marks a language generated at runtime by
	// grammargen (extension grammars registered via RegisterExtension);
	// these have no embedded blob.
	GrammarSourceGrammargen GrammarSource = "grammargen"
	// GrammarSourceGrammargenBlob marks an embedded .bin blob compiled by
	// grammargen (cmd/grammargen -lr-split -bin ...), e.g. go.bin.
	GrammarSourceGrammargenBlob GrammarSource = "grammargen_blob"
)

type HTMLExternalScanner 

type HTMLExternalScanner struct{}

HTMLExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-html.

This is a Go port of the C external scanner from tree-sitter-html (https://github.com/tree-sitter/tree-sitter-html). It reuses the shared HTML scanning infrastructure (html_tags.go, blade_scanner.go) that is also used by the blade, svelte, vue, angular, and astro scanners.

func (HTMLExternalScanner) Create 

func (HTMLExternalScanner) Create() any

func (HTMLExternalScanner) Deserialize 

func (HTMLExternalScanner) Deserialize(payload any, buf []byte)

func (HTMLExternalScanner) Destroy 

func (HTMLExternalScanner) Destroy(payload any)

func (HTMLExternalScanner) Scan 

func (HTMLExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HTMLExternalScanner) Serialize 

func (HTMLExternalScanner) Serialize(payload any, buf []byte) int

type HTMLTokenSource 

type HTMLTokenSource struct {
	// contains filtered or unexported fields
}

HTMLTokenSource is a lightweight lexer bridge for tree-sitter-html. It targets standard tag/text/comment flows and is intended for parser coverage and editor use-cases.

func NewHTMLTokenSource 

func NewHTMLTokenSource(src []byte, lang *gotreesitter.Language) (*HTMLTokenSource, error)

NewHTMLTokenSource creates a token source for HTML source text.

func (*HTMLTokenSource) Next 

func (ts *HTMLTokenSource) Next() gotreesitter.Token

func (*HTMLTokenSource) Reset 

func (ts *HTMLTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*HTMLTokenSource) SkipToByte 

func (ts *HTMLTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*HTMLTokenSource) SupportsIncrementalReuse 

func (ts *HTMLTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that HTMLTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type HackExternalScanner 

type HackExternalScanner struct{}

HackExternalScanner handles heredoc/nowdoc strings for Hack.

func (HackExternalScanner) Create 

func (HackExternalScanner) Create() any

func (HackExternalScanner) Deserialize 

func (HackExternalScanner) Deserialize(payload any, buf []byte)

func (HackExternalScanner) Destroy 

func (HackExternalScanner) Destroy(payload any)

func (HackExternalScanner) Scan 

func (HackExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HackExternalScanner) Serialize 

func (HackExternalScanner) Serialize(payload any, buf []byte) int

type HaskellExternalScanner 

type HaskellExternalScanner struct{}

HaskellExternalScanner implements the gotreesitter.ExternalScanner interface.

func (HaskellExternalScanner) Create 

func (HaskellExternalScanner) Create() any

func (HaskellExternalScanner) Deserialize 

func (HaskellExternalScanner) Deserialize(payload any, buf []byte)

func (HaskellExternalScanner) Destroy 

func (HaskellExternalScanner) Destroy(_ any)

func (HaskellExternalScanner) Scan 

func (HaskellExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HaskellExternalScanner) Serialize 

func (HaskellExternalScanner) Serialize(payload any, buf []byte) int

type HaxeExternalScanner 

type HaxeExternalScanner struct{}

HaxeExternalScanner handles lookback semicolons and closing brace detection for Haxe automatic semicolon insertion.

func (HaxeExternalScanner) Create 

func (HaxeExternalScanner) Create() any

func (HaxeExternalScanner) Deserialize 

func (HaxeExternalScanner) Deserialize(payload any, buf []byte)

func (HaxeExternalScanner) Destroy 

func (HaxeExternalScanner) Destroy(payload any)

func (HaxeExternalScanner) Scan 

func (HaxeExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HaxeExternalScanner) Serialize 

func (HaxeExternalScanner) Serialize(payload any, buf []byte) int

type HclExternalScanner 

type HclExternalScanner struct {
	// contains filtered or unexported fields
}

HclExternalScanner implements gotreesitter.ExternalScanner for the HCL grammar.

func (HclExternalScanner) Create 

func (HclExternalScanner) Create() any

func (HclExternalScanner) Deserialize 

func (HclExternalScanner) Deserialize(payload any, buf []byte)

func (HclExternalScanner) Destroy 

func (HclExternalScanner) Destroy(payload any)

func (HclExternalScanner) ExternalScannerForLanguage 

func (HclExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

ExternalScannerForLanguage binds this scanner's token slots to lang's external symbols positionally (external index i -> scanner token i), so Scan resolves result symbols through the bound table instead of hardcoded absolute IDs.

func (HclExternalScanner) Scan 

func (scanner HclExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HclExternalScanner) Serialize 

func (HclExternalScanner) Serialize(payload any, buf []byte) int

type HlslExternalScanner 

type HlslExternalScanner struct{}

HlslExternalScanner handles C++ R"delim(...)delim" raw string literals for HLSL.

func (HlslExternalScanner) Create 

func (HlslExternalScanner) Create() any

func (HlslExternalScanner) Deserialize 

func (HlslExternalScanner) Deserialize(payload any, buf []byte)

func (HlslExternalScanner) Destroy 

func (HlslExternalScanner) Destroy(payload any)

func (HlslExternalScanner) Scan 

func (HlslExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (HlslExternalScanner) Serialize 

func (HlslExternalScanner) Serialize(payload any, buf []byte) int

type JSONTokenSource 

type JSONTokenSource struct {
	// contains filtered or unexported fields
}

JSONTokenSource bridges raw JSON source to tree-sitter token symbols. It emits punctuation/literal tokens directly and splits strings into: open quote, string_content / escape_sequence chunks, close quote.

func NewJSONTokenSource 

func NewJSONTokenSource(src []byte, lang *gotreesitter.Language) (*JSONTokenSource, error)

NewJSONTokenSource creates a token source for JSON.

func (*JSONTokenSource) Next 

func (ts *JSONTokenSource) Next() gotreesitter.Token

func (*JSONTokenSource) Reset 

func (ts *JSONTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*JSONTokenSource) SkipToByte 

func (ts *JSONTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*JSONTokenSource) SkipToByteWithPoint 

func (ts *JSONTokenSource) SkipToByteWithPoint(offset uint32, _ gotreesitter.Point) gotreesitter.Token

func (*JSONTokenSource) SupportsIncrementalReuse 

func (ts *JSONTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that JSONTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type JanetExternalScanner 

type JanetExternalScanner struct{}

JanetExternalScanner handles @`...` long buffers and `...` long strings for Janet.

func (JanetExternalScanner) Create 

func (JanetExternalScanner) Create() any

func (JanetExternalScanner) Deserialize 

func (JanetExternalScanner) Deserialize(payload any, buf []byte)

func (JanetExternalScanner) Destroy 

func (JanetExternalScanner) Destroy(payload any)

func (JanetExternalScanner) Scan 

func (JanetExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JanetExternalScanner) Serialize 

func (JanetExternalScanner) Serialize(payload any, buf []byte) int

type JavaScriptExternalScanner 

type JavaScriptExternalScanner struct{}

JavaScriptExternalScanner handles automatic semicolons, template strings, JSX text, ternary question marks, and HTML comments for JavaScript.

func (JavaScriptExternalScanner) Create 

func (JavaScriptExternalScanner) Create() any

func (JavaScriptExternalScanner) Deserialize 

func (JavaScriptExternalScanner) Deserialize(payload any, buf []byte)

func (JavaScriptExternalScanner) Destroy 

func (JavaScriptExternalScanner) Destroy(payload any)

func (JavaScriptExternalScanner) Scan 

func (JavaScriptExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JavaScriptExternalScanner) Serialize 

func (JavaScriptExternalScanner) Serialize(payload any, buf []byte) int

func (JavaScriptExternalScanner) SupportsIncrementalReuse 

func (JavaScriptExternalScanner) SupportsIncrementalReuse() bool

type JavaTokenSource 

type JavaTokenSource struct {
	// contains filtered or unexported fields
}

JavaTokenSource is a lexer bridge for tree-sitter-java.

func NewJavaTokenSource 

func NewJavaTokenSource(src []byte, lang *gotreesitter.Language) (*JavaTokenSource, error)

NewJavaTokenSource creates a token source for Java source text.

func (*JavaTokenSource) Next 

func (ts *JavaTokenSource) Next() gotreesitter.Token

func (*JavaTokenSource) Reset 

func (ts *JavaTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*JavaTokenSource) SetGLRStates 

func (ts *JavaTokenSource) SetGLRStates(states []gotreesitter.StateID)

SetGLRStates lets the token source consider all active GLR branches when deciding whether a contextual keyword is valid.

func (*JavaTokenSource) SetParserState 

func (ts *JavaTokenSource) SetParserState(state gotreesitter.StateID)

SetParserState lets the parser drive contextual keyword tokenization.

func (*JavaTokenSource) SkipToByte 

func (ts *JavaTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*JavaTokenSource) SupportsIncrementalReuse 

func (ts *JavaTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that JavaTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type JsdocExternalScanner 

type JsdocExternalScanner struct{}

JsdocExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-jsdoc.

The jsdoc grammar uses an external scanner to match the "type" token, which represents text inside a JSDoc type annotation between { and }. The scanner tracks brace nesting depth and consumes all content until the unmatched closing } is found.

func (JsdocExternalScanner) Create 

func (JsdocExternalScanner) Create() any

func (JsdocExternalScanner) Deserialize 

func (JsdocExternalScanner) Deserialize(payload any, buf []byte)

func (JsdocExternalScanner) Destroy 

func (JsdocExternalScanner) Destroy(payload any)

func (JsdocExternalScanner) Scan 

func (JsdocExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JsdocExternalScanner) Serialize 

func (JsdocExternalScanner) Serialize(payload any, buf []byte) int

type JsonnetExternalScanner 

type JsonnetExternalScanner struct{}

JsonnetExternalScanner handles Jsonnet string literals including single/double quoted strings and ||| block strings.

func (JsonnetExternalScanner) Create 

func (JsonnetExternalScanner) Create() any

func (JsonnetExternalScanner) Deserialize 

func (JsonnetExternalScanner) Deserialize(payload any, buf []byte)

func (JsonnetExternalScanner) Destroy 

func (JsonnetExternalScanner) Destroy(payload any)

func (JsonnetExternalScanner) Scan 

func (JsonnetExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JsonnetExternalScanner) Serialize 

func (JsonnetExternalScanner) Serialize(payload any, buf []byte) int

type JuliaExternalScanner 

type JuliaExternalScanner struct{}

JuliaExternalScanner handles block comments, immediate tokens, and string/command content for Julia.

func (JuliaExternalScanner) Create 

func (JuliaExternalScanner) Create() any

func (JuliaExternalScanner) Deserialize 

func (JuliaExternalScanner) Deserialize(payload any, buf []byte)

func (JuliaExternalScanner) Destroy 

func (JuliaExternalScanner) Destroy(payload any)

func (JuliaExternalScanner) Scan 

func (JuliaExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JuliaExternalScanner) Serialize 

func (JuliaExternalScanner) Serialize(payload any, buf []byte) int

type JustExternalScanner 

type JustExternalScanner struct{}

JustExternalScanner handles indent/dedent/newline/text for justfiles.

func (JustExternalScanner) Create 

func (JustExternalScanner) Create() any

func (JustExternalScanner) Deserialize 

func (JustExternalScanner) Deserialize(payload any, buf []byte)

func (JustExternalScanner) Destroy 

func (JustExternalScanner) Destroy(payload any)

func (JustExternalScanner) Scan 

func (JustExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (JustExternalScanner) Serialize 

func (JustExternalScanner) Serialize(payload any, buf []byte) int

type KconfigExternalScanner 

type KconfigExternalScanner struct{}

KconfigExternalScanner handles indented help text blocks in Linux Kconfig files. Help text continues as long as lines have consistent indentation.

func (KconfigExternalScanner) Create 

func (KconfigExternalScanner) Create() any

func (KconfigExternalScanner) Deserialize 

func (KconfigExternalScanner) Deserialize(payload any, buf []byte)

func (KconfigExternalScanner) Destroy 

func (KconfigExternalScanner) Destroy(payload any)

func (KconfigExternalScanner) Scan 

func (KconfigExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (KconfigExternalScanner) Serialize 

func (KconfigExternalScanner) Serialize(payload any, buf []byte) int

type KdlExternalScanner 

type KdlExternalScanner struct{}

KdlExternalScanner handles EOF detection, nestable /* */ comments, and r#"..."# raw strings for KDL.

func (KdlExternalScanner) Create 

func (KdlExternalScanner) Create() any

func (KdlExternalScanner) Deserialize 

func (KdlExternalScanner) Deserialize(payload any, buf []byte)

func (KdlExternalScanner) Destroy 

func (KdlExternalScanner) Destroy(payload any)

func (KdlExternalScanner) Scan 

func (KdlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (KdlExternalScanner) Serialize 

func (KdlExternalScanner) Serialize(payload any, buf []byte) int

type KotlinExternalScanner 

type KotlinExternalScanner struct {
	// contains filtered or unexported fields
}

KotlinExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-kotlin.

This is a Go port of the C external scanner from fwcd/tree-sitter-kotlin. The scanner handles 9 external tokens including automatic semicolon insertion (ASI), safe navigation (?.), nested multiline comments, string start/end/content with interpolation support, primary constructor keyword detection, and import path handling.

func (KotlinExternalScanner) Create 

func (KotlinExternalScanner) Create() any

func (KotlinExternalScanner) Deserialize 

func (KotlinExternalScanner) Deserialize(payload any, buf []byte)

func (KotlinExternalScanner) Destroy 

func (KotlinExternalScanner) Destroy(payload any)

func (KotlinExternalScanner) ExternalScannerForLanguage 

func (KotlinExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

func (KotlinExternalScanner) Scan 

func (k KotlinExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (KotlinExternalScanner) Serialize 

func (KotlinExternalScanner) Serialize(payload any, buf []byte) int

type LangEntry 

type LangEntry struct {
	Name               string
	Extensions         []string                      // e.g. [".go", ".mod"]
	Shebangs           []string                      // e.g. ["#!/usr/bin/env python"]
	Language           func() *gotreesitter.Language // lazy loader
	GrammarSource      GrammarSource                 // e.g. ts2go blob or grammargen
	HighlightQuery     string
	InheritHighlights  string                                                                 // language name to inherit highlight queries from (e.g. "javascript" for TypeScript)
	TagsQuery          string                                                                 // tree-sitter tags.scm query for symbol extraction
	TokenSourceFactory func(src []byte, lang *gotreesitter.Language) gotreesitter.TokenSource // nil = use DFA
	Quality            ParseQuality                                                           // populated by AuditParseSupport
}

LangEntry holds a registered language with its grammar, extensions, and highlight query.

func AllLanguages 

func AllLanguages() []LangEntry

AllLanguages returns all registered languages. This is a metadata-only operation — it does NOT load grammar parse tables or decompress blobs. Languages that lack an explicit TagsQuery will have an empty TagsQuery field; call ResolveTagsQuery when you actually need the inferred query.

func DetectLanguage 

func DetectLanguage(filename string) *LangEntry

DetectLanguage returns the LangEntry for a filename, or nil if unknown. Checks in order: exact filename match (linguist), registry extensions, then linguist extended extensions. Exact filenames take priority over suffix matching so that e.g. ".tmux.conf" resolves to bash rather than matching the generic ".conf" extension.

func DetectLanguageByName 

func DetectLanguageByName(name string) *LangEntry

DetectLanguageByName returns the LangEntry for any linguist canonical name, alias, or gotreesitter grammar name. Returns nil if unknown.

Accepts: "C++", "cpp", "Go", "golang", "Shell", "bash", "F#", "fsharp", etc. Direct grammar names always take priority over linguist aliases to prevent shadowing (e.g., "eex" resolves to the eex grammar, not heex via alias).

func DetectLanguageByShebang 

func DetectLanguageByShebang(firstLine string) *LangEntry

DetectLanguageByShebang checks the first line of content for shebang matches. Handles both "#!/usr/bin/env python3" and "#!/usr/bin/python3" forms.

func DetectLanguageWithContent 

func DetectLanguageWithContent(filename string, content []byte) *LangEntry

DetectLanguageWithContent resolves a LangEntry the same way DetectLanguage does, but additionally sniffs file content to disambiguate a plain ".h" extension between the C and C++ grammars.

Background: gotreesitter's extension table maps both the C and the C++ LangEntry to bare ".h" (mirroring the real world — most ".h" files are C, but plenty of C++ libraries ship C++-only headers under a ".h" name, e.g. ImageMagick's Magick++/*.h, OpenEXR's Imath/*.h). DetectLanguage resolves that tie in favor of C (first grammar registered for the extension; see buildExtIndex), which mis-parses genuine C++ headers with the C grammar.

When content is available — which it always is at parse time, see ParseFile / ParseFilePooled — this function re-checks: if extension-based detection landed on the C grammar AND the extension is exactly ".h" (the one genuinely ambiguous case), it scans content for syntax that is valid C++ and never valid C (see looksLikeCppHeader) and, if found, returns the C++ grammar instead.

Every other extension (.c, .cc, .cpp, .hpp, .hh, .hxx, and every non-C/C++ language) is untouched — this only refines the single ambiguous case. DetectLanguage(filename) itself is unchanged and remains the content-free entry point for callers without a content sniff.

type LessExternalScanner 

type LessExternalScanner struct{}

LessExternalScanner handles the CSS descendant combinator for Less. Nearly identical to the SCSS descendant operator scanner.

func (LessExternalScanner) Create 

func (LessExternalScanner) Create() any

func (LessExternalScanner) Deserialize 

func (LessExternalScanner) Deserialize(payload any, buf []byte)

func (LessExternalScanner) Destroy 

func (LessExternalScanner) Destroy(payload any)

func (LessExternalScanner) Scan 

func (LessExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (LessExternalScanner) Serialize 

func (LessExternalScanner) Serialize(payload any, buf []byte) int

type LiquidExternalScanner 

type LiquidExternalScanner struct{}

LiquidExternalScanner handles comment content, raw blocks, and front matter for Liquid templates.

func (LiquidExternalScanner) Create 

func (LiquidExternalScanner) Create() any

func (LiquidExternalScanner) Deserialize 

func (LiquidExternalScanner) Deserialize(payload any, buf []byte)

func (LiquidExternalScanner) Destroy 

func (LiquidExternalScanner) Destroy(payload any)

func (LiquidExternalScanner) Scan 

func (LiquidExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (LiquidExternalScanner) Serialize 

func (LiquidExternalScanner) Serialize(payload any, buf []byte) int

type LuaExternalScanner 

type LuaExternalScanner struct{}

LuaExternalScanner is a line-faithful port of the pinned upstream src/scanner.c (tree-sitter-grammars/tree-sitter-lua @ 10fe0054). It scans long-bracket block strings/comments: [[ ... ]], [=[ ... ]=], etc.

func (LuaExternalScanner) Create 

func (LuaExternalScanner) Create() any

func (LuaExternalScanner) Deserialize 

func (LuaExternalScanner) Deserialize(payload any, buf []byte)

func (LuaExternalScanner) Destroy 

func (LuaExternalScanner) Destroy(payload any)

func (LuaExternalScanner) Scan 

func (LuaExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (LuaExternalScanner) Serialize 

func (LuaExternalScanner) Serialize(payload any, buf []byte) int

type LuaTokenSource 

type LuaTokenSource struct {
	// contains filtered or unexported fields
}

LuaTokenSource is a lexer bridge for tree-sitter-lua.

func NewLuaTokenSource 

func NewLuaTokenSource(src []byte, lang *gotreesitter.Language) (*LuaTokenSource, error)

NewLuaTokenSource creates a token source for Lua source text.

func (*LuaTokenSource) Next 

func (ts *LuaTokenSource) Next() gotreesitter.Token

func (*LuaTokenSource) Reset 

func (ts *LuaTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*LuaTokenSource) SkipToByte 

func (ts *LuaTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*LuaTokenSource) SupportsIncrementalReuse 

func (ts *LuaTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that LuaTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type LuauExternalScanner 

type LuauExternalScanner struct{}

LuauExternalScanner handles Lua-style block comments --[=[ ... ]=] and block/quoted strings for Luau.

func (LuauExternalScanner) Create 

func (LuauExternalScanner) Create() any

func (LuauExternalScanner) Deserialize 

func (LuauExternalScanner) Deserialize(payload any, buf []byte)

func (LuauExternalScanner) Destroy 

func (LuauExternalScanner) Destroy(payload any)

func (LuauExternalScanner) Scan 

func (LuauExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (LuauExternalScanner) Serialize 

func (LuauExternalScanner) Serialize(payload any, buf []byte) int

type MarkdownExternalScanner 

type MarkdownExternalScanner struct{}

MarkdownExternalScanner handles block-level markdown parsing.

func (MarkdownExternalScanner) Create 

func (MarkdownExternalScanner) Create() any

func (MarkdownExternalScanner) Deserialize 

func (MarkdownExternalScanner) Deserialize(payload any, buf []byte)

func (MarkdownExternalScanner) Destroy 

func (MarkdownExternalScanner) Destroy(payload any)

func (MarkdownExternalScanner) Scan 

func (MarkdownExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (MarkdownExternalScanner) Serialize 

func (MarkdownExternalScanner) Serialize(payload any, buf []byte) int

type MarkdownInlineExternalScanner 

type MarkdownInlineExternalScanner struct{}

MarkdownInlineExternalScanner handles external scanning for the markdown_inline grammar.

func (MarkdownInlineExternalScanner) Create 

func (MarkdownInlineExternalScanner) Create() any

func (MarkdownInlineExternalScanner) Deserialize 

func (MarkdownInlineExternalScanner) Deserialize(payload any, buf []byte)

func (MarkdownInlineExternalScanner) Destroy 

func (MarkdownInlineExternalScanner) Destroy(payload any)

func (MarkdownInlineExternalScanner) Scan 

func (MarkdownInlineExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (MarkdownInlineExternalScanner) Serialize 

func (MarkdownInlineExternalScanner) Serialize(payload any, buf []byte) int

type MatlabExternalScanner 

type MatlabExternalScanner struct{}

MatlabExternalScanner handles the external scanning for the MATLAB grammar.

func (MatlabExternalScanner) Create 

func (MatlabExternalScanner) Create() any

func (MatlabExternalScanner) Deserialize 

func (MatlabExternalScanner) Deserialize(payload any, buf []byte)

func (MatlabExternalScanner) Destroy 

func (MatlabExternalScanner) Destroy(payload any)

func (MatlabExternalScanner) Scan 

func (MatlabExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (MatlabExternalScanner) Serialize 

func (MatlabExternalScanner) Serialize(payload any, buf []byte) int

type MojoExternalScanner 

type MojoExternalScanner struct{}

MojoExternalScanner is the tree-sitter-python external scanner retargeted at the mojo grammar's symbol IDs. Mojo's scanner.c is byte-for-byte python's scanner minus the `except` external, so the state shape and the scan logic are shared with PythonExternalScanner.

func (MojoExternalScanner) Create 

func (MojoExternalScanner) Create() any

func (MojoExternalScanner) Deserialize 

func (MojoExternalScanner) Deserialize(payload any, buf []byte)

func (MojoExternalScanner) Destroy 

func (MojoExternalScanner) Destroy(payload any)

func (MojoExternalScanner) Scan 

func (MojoExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (MojoExternalScanner) Serialize 

func (MojoExternalScanner) Serialize(payload any, buf []byte) int

func (MojoExternalScanner) SupportsIncrementalReuse 

func (MojoExternalScanner) SupportsIncrementalReuse() bool

type MoveExternalScanner 

type MoveExternalScanner struct{}

MoveExternalScanner ports the stateless upstream scanner: block doc-comment markers (`/**` but not `/***` or `/**/`), nestable block comment content, and doc line comment bodies (`/// ...` up to and including EOL).

func (MoveExternalScanner) Create 

func (MoveExternalScanner) Create() any

func (MoveExternalScanner) Deserialize 

func (MoveExternalScanner) Deserialize(payload any, buf []byte)

func (MoveExternalScanner) Destroy 

func (MoveExternalScanner) Destroy(payload any)

func (MoveExternalScanner) Scan 

func (MoveExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (MoveExternalScanner) Serialize 

func (MoveExternalScanner) Serialize(payload any, buf []byte) int

type NginxExternalScanner 

type NginxExternalScanner struct{}

NginxExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-nginx.

This is a Go port of the C external scanner from tree-sitter-nginx (https://github.com/opa-oz/tree-sitter-nginx). The scanner handles:

  • _newline: newline characters
  • _indent: increase in indentation level
  • _dedent: decrease in indentation level

func (NginxExternalScanner) Create 

func (NginxExternalScanner) Create() any

func (NginxExternalScanner) Deserialize 

func (NginxExternalScanner) Deserialize(payload any, buf []byte)

func (NginxExternalScanner) Destroy 

func (NginxExternalScanner) Destroy(payload any)

func (NginxExternalScanner) Scan 

func (NginxExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NginxExternalScanner) Serialize 

func (NginxExternalScanner) Serialize(payload any, buf []byte) int

type NickelExternalScanner 

type NickelExternalScanner struct{}

NickelExternalScanner handles Nickel's multi-line strings, interpolation, and comments.

func (NickelExternalScanner) Create 

func (NickelExternalScanner) Create() any

func (NickelExternalScanner) Deserialize 

func (NickelExternalScanner) Deserialize(payload any, buf []byte)

func (NickelExternalScanner) Destroy 

func (NickelExternalScanner) Destroy(payload any)

func (NickelExternalScanner) Scan 

func (NickelExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NickelExternalScanner) Serialize 

func (NickelExternalScanner) Serialize(payload any, buf []byte) int

type NimExternalScanner 

type NimExternalScanner struct{}

--------------------------------------------------------------------------- NimExternalScanner implements gotreesitter.ExternalScanner ---------------------------------------------------------------------------

func (NimExternalScanner) Create 

func (NimExternalScanner) Create() any

func (NimExternalScanner) Deserialize 

func (NimExternalScanner) Deserialize(payload any, buf []byte)

func (NimExternalScanner) Destroy 

func (NimExternalScanner) Destroy(payload any)

func (NimExternalScanner) Scan 

func (NimExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NimExternalScanner) Serialize 

func (NimExternalScanner) Serialize(payload any, buf []byte) int

type NixExternalScanner 

type NixExternalScanner struct{}

NixExternalScanner handles string fragments, path literals, and dollar escapes for the Nix expression language.

func (NixExternalScanner) Create 

func (NixExternalScanner) Create() any

func (NixExternalScanner) Deserialize 

func (NixExternalScanner) Deserialize(payload any, buf []byte)

func (NixExternalScanner) Destroy 

func (NixExternalScanner) Destroy(payload any)

func (NixExternalScanner) Scan 

func (NixExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NixExternalScanner) Serialize 

func (NixExternalScanner) Serialize(payload any, buf []byte) int

type NorgExternalScanner 

type NorgExternalScanner struct{}

NorgExternalScanner handles norg markup disambiguation.

func (NorgExternalScanner) Create 

func (NorgExternalScanner) Create() any

func (NorgExternalScanner) Deserialize 

func (NorgExternalScanner) Deserialize(payload any, buf []byte)

func (NorgExternalScanner) Destroy 

func (NorgExternalScanner) Destroy(_ any)

func (NorgExternalScanner) Scan 

func (NorgExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NorgExternalScanner) Serialize 

func (NorgExternalScanner) Serialize(payload any, buf []byte) int

type NushellExternalScanner 

type NushellExternalScanner struct{}

NushellExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-nu.

This is a Go port of the C external scanner from tree-sitter-nu (https://github.com/nushell/tree-sitter-nu). The scanner handles:

  • raw_string_begin: r#' (with variable # count)
  • string_content: content between raw string delimiters
  • raw_string_end: '# closing delimiter

func (NushellExternalScanner) Create 

func (NushellExternalScanner) Create() any

func (NushellExternalScanner) Deserialize 

func (NushellExternalScanner) Deserialize(payload any, buf []byte)

func (NushellExternalScanner) Destroy 

func (NushellExternalScanner) Destroy(payload any)

func (NushellExternalScanner) Scan 

func (NushellExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (NushellExternalScanner) Serialize 

func (NushellExternalScanner) Serialize(payload any, buf []byte) int

type OcamlExternalScanner 

type OcamlExternalScanner struct{}

OcamlExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-ocaml.

This is a Go port of the C external scanner from tree-sitter/tree-sitter-ocaml. The scanner handles:

  • Nestable (* *) comments (lexically aware of strings/chars inside)
  • Quoted string delimiters {id|...|id}
  • String open/close with in_string state tracking
  • Line number directives (# <num> "file")
  • Literal null characters (\0 that isn't EOF)

func (OcamlExternalScanner) Create 

func (OcamlExternalScanner) Create() any

func (OcamlExternalScanner) Deserialize 

func (OcamlExternalScanner) Deserialize(payload any, buf []byte)

func (OcamlExternalScanner) Destroy 

func (OcamlExternalScanner) Destroy(payload any)

func (OcamlExternalScanner) Scan 

func (OcamlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (OcamlExternalScanner) Serialize 

func (OcamlExternalScanner) Serialize(payload any, buf []byte) int

type OdinExternalScanner 

type OdinExternalScanner struct{}

OdinExternalScanner handles newlines, floats, block comments, and other context-sensitive tokens for Odin.

func (OdinExternalScanner) Create 

func (OdinExternalScanner) Create() any

func (OdinExternalScanner) Deserialize 

func (OdinExternalScanner) Deserialize(payload any, buf []byte)

func (OdinExternalScanner) Destroy 

func (OdinExternalScanner) Destroy(payload any)

func (OdinExternalScanner) Scan 

func (OdinExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (OdinExternalScanner) Serialize 

func (OdinExternalScanner) Serialize(payload any, buf []byte) int

type OrgExternalScanner 

type OrgExternalScanner struct{}

OrgExternalScanner handles list/section/headline detection for org mode.

func (OrgExternalScanner) Create 

func (OrgExternalScanner) Create() any

func (OrgExternalScanner) Deserialize 

func (OrgExternalScanner) Deserialize(payload any, buf []byte)

func (OrgExternalScanner) Destroy 

func (OrgExternalScanner) Destroy(payload any)

func (OrgExternalScanner) Scan 

func (OrgExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (OrgExternalScanner) Serialize 

func (OrgExternalScanner) Serialize(payload any, buf []byte) int

type ParseBackend 

type ParseBackend string

ParseBackend describes how a language can be parsed in this runtime. 

const (
	ParseBackendUnsupported ParseBackend = "unsupported"
	ParseBackendDFA         ParseBackend = "dfa"
	ParseBackendDFAPartial  ParseBackend = "dfa-partial"
	ParseBackendTokenSource ParseBackend = "token_source"
)

type ParsePolicy 

type ParsePolicy struct {
	// LargeFileThreshold is the byte size above which a file is parsed with
	// exclusive access to all worker slots (serialized). Files at or above
	// this threshold are considered "large."
	LargeFileThreshold int64

	// MaxConcurrent limits the number of files parsed simultaneously.
	MaxConcurrent int

	// ChannelBuffer is the buffer size for the output channel. Must be at
	// least MaxConcurrent+1 to prevent deadlock when workers hold semaphore
	// slots while sending on the channel.
	ChannelBuffer int

	// SkipDirs lists directory base names to skip entirely during the walk.
	SkipDirs []string

	// ShouldSkipDir, if non-nil, is called for each directory before descent.
	// The root directory is never skipped. Return true to skip the directory
	// and all descendants.
	ShouldSkipDir func(path string) bool

	// SkipExtensions lists file suffixes (e.g., ".min.js") to skip.
	SkipExtensions []string

	// ShouldParse, if non-nil, is called for each candidate file after
	// language detection. Return false to skip the file.
	ShouldParse func(path string, size int64, modTime time.Time) bool

	// SkipTreeParse, if non-nil, is called for candidate files that pass
	// ShouldParse. Return true to read the file (populating Source) but
	// skip the tree-sitter parse (Tree will be nil, Err will be nil).
	// This is useful for large generated files where the consumer wants
	// the raw source bytes for lightweight extraction without paying for
	// a full AST parse.
	SkipTreeParse func(path string, size int64) bool

	// OnProgress, if non-nil, receives progress events during the walk.
	OnProgress func(ProgressEvent)
}

ParsePolicy controls how WalkAndParse discovers and parses files. Configure concurrency limits, file size thresholds, directory/extension filters, and optional hooks for progress reporting and file filtering. Use DefaultPolicy to get a policy with sensible defaults.

func DefaultPolicy 

func DefaultPolicy() ParsePolicy

DefaultPolicy returns a ParsePolicy with sensible defaults for typical source repositories:

  • LargeFileThreshold: 256 KB (env: GTS_LARGE_FILE_THRESHOLD, in bytes)
  • MaxConcurrent: runtime.GOMAXPROCS(0) (env: GTS_MAX_CONCURRENT)
  • ChannelBuffer: MaxConcurrent + 1
  • SkipDirs: .git, .graft, .hg, .svn, vendor, node_modules
  • SkipExtensions: .min.js, .min.css, .map, .wasm

Environment variables are read on each call and must be positive integers. Invalid or non-positive values are silently ignored (defaults apply).

type ParseQuality 

type ParseQuality string

ParseQuality summarizes how trustworthy a grammar's parse output is. 

const (
	ParseQualityFull    ParseQuality = "full"    // token_source or dfa with scanner
	ParseQualityPartial ParseQuality = "partial" // dfa-partial (missing ext scanner)
	ParseQualityNone    ParseQuality = "none"    // cannot parse
)

type ParseSupport 

type ParseSupport struct {
	Name                    string
	LanguageVersion         uint32
	VersionCompatible       bool
	Backend                 ParseBackend
	Reason                  string
	HasTokenSourceFactory   bool
	HasDFALexer             bool
	RequiresExternalScanner bool
	HasExternalScanner      bool
}

ParseSupport summarizes parser support status for one registered language.

func AuditParseSupport 

func AuditParseSupport() []ParseSupport

AuditParseSupport evaluates parse support for all registered languages.

func EvaluateParseSupport 

func EvaluateParseSupport(entry LangEntry, lang *gotreesitter.Language) ParseSupport

EvaluateParseSupport reports whether a language can parse using either the built-in DFA lexer or a registered custom token source factory.

func ParseSupportFor 

func ParseSupportFor(name string) (ParseSupport, bool)

ParseSupportFor evaluates one registered language without loading the rest of the grammar fleet. It returns false when name is not a canonical registry name or the language loader returns nil.

type ParsedFile 

type ParsedFile struct {
	Path   string                  // absolute path to the source file
	Tree   *gotreesitter.BoundTree // parsed AST; nil on error. Consumer MUST call Close()
	Lang   *LangEntry              // detected language entry
	Source []byte                  // raw file contents; nil on I/O error
	Size   int64                   // file size in bytes (from stat, before read)
	Err    error                   // nil on success
	IsRead bool                    // false = I/O error during read, true = file was read (parse may have failed)
}

ParsedFile is a single result from WalkAndParse. Each ParsedFile owns its BoundTree and Source slice. The consumer MUST call ParsedFile.Close when finished inspecting the tree to release arena memory. Failing to close results will leak memory proportional to the parsed file sizes.

When Err is non-nil, check IsRead to distinguish I/O errors (IsRead=false, file could not be read) from parse errors (IsRead=true, file was read but the grammar could not parse it). On I/O errors, Source and Tree are nil.

func (*ParsedFile) Close 

func (pf *ParsedFile) Close()

Close releases the BoundTree's arena memory and nils both Tree and Source to allow garbage collection. Close is safe to call multiple times and on a nil receiver.

Ownership rule: the consumer that receives a ParsedFile from the channel owns it. Once Close is called, Tree becomes nil and must not be used. Always defer Close or call it explicitly after processing each result: 

for pf := range ch {
    if pf.Err != nil { handleErr(pf); pf.Close(); continue }
    process(pf.Tree, pf.Source)
    pf.Close()
}

type PerlExternalScanner 

type PerlExternalScanner struct{}

PerlExternalScanner implements gotreesitter.ExternalScanner for Perl.

func (PerlExternalScanner) Create 

func (PerlExternalScanner) Create() any

func (PerlExternalScanner) Deserialize 

func (PerlExternalScanner) Deserialize(payload any, buf []byte)

func (PerlExternalScanner) Destroy 

func (PerlExternalScanner) Destroy(payload any)

func (PerlExternalScanner) Scan 

func (PerlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PerlExternalScanner) Serialize 

func (PerlExternalScanner) Serialize(payload any, buf []byte) int

type PhpExternalScanner 

type PhpExternalScanner struct{}

PhpExternalScanner handles heredocs, encapsed strings, auto-semicolons, and EOF for PHP.

func (PhpExternalScanner) Create 

func (PhpExternalScanner) Create() any

func (PhpExternalScanner) Deserialize 

func (PhpExternalScanner) Deserialize(payload any, buf []byte)

func (PhpExternalScanner) Destroy 

func (PhpExternalScanner) Destroy(payload any)

func (PhpExternalScanner) Scan 

func (PhpExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PhpExternalScanner) Serialize 

func (PhpExternalScanner) Serialize(payload any, buf []byte) int

type PklExternalScanner 

type PklExternalScanner struct{}

PklExternalScanner handles string content and contextual operators for Pkl.

func (PklExternalScanner) Create 

func (PklExternalScanner) Create() any

func (PklExternalScanner) Deserialize 

func (PklExternalScanner) Deserialize(payload any, buf []byte)

func (PklExternalScanner) Destroy 

func (PklExternalScanner) Destroy(payload any)

func (PklExternalScanner) Scan 

func (PklExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PklExternalScanner) Serialize 

func (PklExternalScanner) Serialize(payload any, buf []byte) int

type PowershellExternalScanner 

type PowershellExternalScanner struct{}

PowershellExternalScanner handles statement termination detection for PowerShell. A statement terminator is a zero-width token that fires when the next significant character is EOF, }, ;, ), or newline.

func (PowershellExternalScanner) Create 

func (PowershellExternalScanner) Create() any

func (PowershellExternalScanner) Deserialize 

func (PowershellExternalScanner) Deserialize(payload any, buf []byte)

func (PowershellExternalScanner) Destroy 

func (PowershellExternalScanner) Destroy(payload any)

func (PowershellExternalScanner) Scan 

func (PowershellExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PowershellExternalScanner) Serialize 

func (PowershellExternalScanner) Serialize(payload any, buf []byte) int

type ProgressEvent 

type ProgressEvent struct {
	Phase   string // one of: "walking", "parsing", "large_file", "walk_complete", "done"
	Path    string // file path (empty for walk_complete and done phases)
	Size    int64  // file size in bytes
	FileNum int    // 1-based ordinal of this file among discovered files
	Total   int    // total files found (set only in walk_complete phase)
	Message string // optional human-readable detail
}

ProgressEvent reports progress during WalkAndParse. The Phase field indicates the stage of processing:

  • "walking" — a file was discovered and queued for parsing
  • "parsing" — a file is being parsed (normal or large)
  • "large_file" — a large file is acquiring exclusive access
  • "walk_complete" — directory walk finished; Total is set
  • "done" — all parsing complete, channel about to close

type PropertiesExternalScanner 

type PropertiesExternalScanner struct{}

PropertiesExternalScanner handles EOF detection for Java .properties files.

func (PropertiesExternalScanner) Create 

func (PropertiesExternalScanner) Create() any

func (PropertiesExternalScanner) Deserialize 

func (PropertiesExternalScanner) Deserialize(payload any, buf []byte)

func (PropertiesExternalScanner) Destroy 

func (PropertiesExternalScanner) Destroy(payload any)

func (PropertiesExternalScanner) Scan 

func (PropertiesExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PropertiesExternalScanner) Serialize 

func (PropertiesExternalScanner) Serialize(payload any, buf []byte) int

type PugExternalScanner 

type PugExternalScanner struct{}

PugExternalScanner handles newline/indent/dedent for Pug templates.

func (PugExternalScanner) Create 

func (PugExternalScanner) Create() any

func (PugExternalScanner) Deserialize 

func (PugExternalScanner) Deserialize(payload any, buf []byte)

func (PugExternalScanner) Destroy 

func (PugExternalScanner) Destroy(payload any)

func (PugExternalScanner) Scan 

func (PugExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PugExternalScanner) Serialize 

func (PugExternalScanner) Serialize(payload any, buf []byte) int

type PurescriptExternalScanner 

type PurescriptExternalScanner struct{}

PurescriptExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-purescript.

This is a Go port of the C external scanner from tree-sitter-purescript (https://github.com/purerl/tree-sitter-purescript). The scanner handles layout-sensitive indentation for PureScript's offside rule, tracking an indent stack and emitting layout tokens (semicolons, starts, ends), as well as qualified module dots, comments, and various keyword tokens.

func (PurescriptExternalScanner) Create 

func (PurescriptExternalScanner) Create() any

func (PurescriptExternalScanner) Deserialize 

func (PurescriptExternalScanner) Deserialize(payload any, buf []byte)

func (PurescriptExternalScanner) Destroy 

func (PurescriptExternalScanner) Destroy(payload any)

func (PurescriptExternalScanner) Scan 

func (PurescriptExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PurescriptExternalScanner) Serialize 

func (PurescriptExternalScanner) Serialize(payload any, buf []byte) int

type PythonExternalScanner 

type PythonExternalScanner struct {
	// contains filtered or unexported fields
}

func (PythonExternalScanner) Create 

func (PythonExternalScanner) Create() any

func (PythonExternalScanner) Deserialize 

func (PythonExternalScanner) Deserialize(payload any, buf []byte)

func (PythonExternalScanner) Destroy 

func (PythonExternalScanner) Destroy(payload any)

func (PythonExternalScanner) ExternalScannerForLanguage 

func (PythonExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

func (PythonExternalScanner) PreservesStateOnScanFailure 

func (PythonExternalScanner) PreservesStateOnScanFailure() bool

func (PythonExternalScanner) Scan 

func (p PythonExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (PythonExternalScanner) Serialize 

func (PythonExternalScanner) Serialize(payload any, buf []byte) int

func (PythonExternalScanner) SupportsIncrementalReuse 

func (PythonExternalScanner) SupportsIncrementalReuse() bool

type RExternalScanner 

type RExternalScanner struct{}

RExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-r.

This is a Go port of the C external scanner from tree-sitter-r (https://github.com/r-lib/tree-sitter-r). The scanner handles:

  • _start: zero-width token emitted at the beginning of the file
  • _newline: contextual newlines in top-level and brace scopes
  • _semicolon: semicolons
  • _raw_string_literal: R raw string literals (r"(...)", R"[...]", etc.)
  • else: the 'else' keyword with special newline handling in brace scopes
  • bracket/brace/paren: scope tracking for (, ), {, }, [, ], [[, ]]
  • _error_sentinel: error recovery detection

func (RExternalScanner) Create 

func (RExternalScanner) Create() any

func (RExternalScanner) Deserialize 

func (RExternalScanner) Deserialize(payload any, buf []byte)

func (RExternalScanner) Destroy 

func (RExternalScanner) Destroy(payload any)

func (RExternalScanner) Scan 

func (RExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RExternalScanner) Serialize 

func (RExternalScanner) Serialize(payload any, buf []byte) int

type RacketExternalScanner 

type RacketExternalScanner struct{}

RacketExternalScanner handles #<<DELIM here-strings for Racket. The scanner reads the terminator from the first line, then consumes subsequent lines until it finds one matching the terminator exactly.

func (RacketExternalScanner) Create 

func (RacketExternalScanner) Create() any

func (RacketExternalScanner) Deserialize 

func (RacketExternalScanner) Deserialize(payload any, buf []byte)

func (RacketExternalScanner) Destroy 

func (RacketExternalScanner) Destroy(payload any)

func (RacketExternalScanner) Scan 

func (RacketExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RacketExternalScanner) Serialize 

func (RacketExternalScanner) Serialize(payload any, buf []byte) int

type RescriptExternalScanner 

type RescriptExternalScanner struct{}

RescriptExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-rescript.

This is a Go port of the C external scanner from rescript-lang/tree-sitter-rescript. The scanner handles 12 external tokens: newlines (with statement-termination semantics), line and block comments, template string characters, string delimiters (" and `), parenthesis nesting, list/dict constructors, and decorator identifiers.

func (RescriptExternalScanner) Create 

func (RescriptExternalScanner) Create() any

func (RescriptExternalScanner) Deserialize 

func (RescriptExternalScanner) Deserialize(payload any, buf []byte)

func (RescriptExternalScanner) Destroy 

func (RescriptExternalScanner) Destroy(payload any)

func (RescriptExternalScanner) Scan 

func (RescriptExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RescriptExternalScanner) Serialize 

func (RescriptExternalScanner) Serialize(payload any, buf []byte) int

type RonExternalScanner 

type RonExternalScanner struct{}

RonExternalScanner handles string content, raw strings, floats, and nestable block comments for RON (Rusty Object Notation).

func (RonExternalScanner) Create 

func (RonExternalScanner) Create() any

func (RonExternalScanner) Deserialize 

func (RonExternalScanner) Deserialize(payload any, buf []byte)

func (RonExternalScanner) Destroy 

func (RonExternalScanner) Destroy(payload any)

func (RonExternalScanner) Scan 

func (RonExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RonExternalScanner) Serialize 

func (RonExternalScanner) Serialize(payload any, buf []byte) int

type RstExternalScanner 

type RstExternalScanner struct{}

func (RstExternalScanner) Create 

func (RstExternalScanner) Create() any

func (RstExternalScanner) Deserialize 

func (RstExternalScanner) Deserialize(payload any, buf []byte)

func (RstExternalScanner) Destroy 

func (RstExternalScanner) Destroy(payload any)

func (RstExternalScanner) Scan 

func (RstExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RstExternalScanner) Serialize 

func (RstExternalScanner) Serialize(payload any, buf []byte) int

type RubyExternalScanner 

type RubyExternalScanner struct{}

RubyExternalScanner handles all external tokens for the Ruby grammar.

func (RubyExternalScanner) Create 

func (RubyExternalScanner) Create() any

func (RubyExternalScanner) Deserialize 

func (RubyExternalScanner) Deserialize(payload any, buf []byte)

func (RubyExternalScanner) Destroy 

func (RubyExternalScanner) Destroy(payload any)

func (RubyExternalScanner) Scan 

func (RubyExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RubyExternalScanner) Serialize 

func (RubyExternalScanner) Serialize(payload any, buf []byte) int

type RustExternalScanner 

type RustExternalScanner struct {
	// contains filtered or unexported fields
}

RustExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-rust.

This is a Go port of the C external scanner from tree-sitter/tree-sitter-rust. The scanner handles 10 external tokens:

  • String content (regular string escape sequences)
  • Raw string literals (r#"..."# with hash counting for start/content/end)
  • Float literals (disambiguation from integer + method call)
  • Block comment content (nested /* */ comments)
  • Doc comments (/// and //!)
  • Outer/inner doc comment markers
  • Error sentinel for error recovery detection

func (RustExternalScanner) Create 

func (RustExternalScanner) Create() any

func (RustExternalScanner) Deserialize 

func (RustExternalScanner) Deserialize(payload any, buf []byte)

func (RustExternalScanner) Destroy 

func (RustExternalScanner) Destroy(payload any)

func (RustExternalScanner) ExternalScannerForLanguage 

func (RustExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

func (RustExternalScanner) Scan 

func (scanner RustExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (RustExternalScanner) Serialize 

func (RustExternalScanner) Serialize(payload any, buf []byte) int

type ScalaExternalScanner 

type ScalaExternalScanner struct{}

ScalaExternalScanner handles auto-semicolons, indent/outdent, and string scanning for Scala.

func (ScalaExternalScanner) Create 

func (ScalaExternalScanner) Create() any

func (ScalaExternalScanner) Deserialize 

func (ScalaExternalScanner) Deserialize(payload any, buf []byte)

func (ScalaExternalScanner) Destroy 

func (ScalaExternalScanner) Destroy(payload any)

func (ScalaExternalScanner) Scan 

func (ScalaExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ScalaExternalScanner) Serialize 

func (ScalaExternalScanner) Serialize(payload any, buf []byte) int

type ScssExternalScanner 

type ScssExternalScanner struct{}

ScssExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-scss.

Ported from tree-sitter-scss/src/scanner.c (pinned commit bca847c). The scanner handles four external tokens:

  • _descendant_operator: whitespace between two selectors (e.g., "div p")
  • _pseudo_class_selector_colon: colon in selector context (e.g., ":hover")
  • __error_recovery: sentinel — always declined
  • _concat: adjacent selector concatenation (e.g., "#foo.bar")

func (ScssExternalScanner) Create 

func (ScssExternalScanner) Create() any

func (ScssExternalScanner) Deserialize 

func (ScssExternalScanner) Deserialize(payload any, buf []byte)

func (ScssExternalScanner) Destroy 

func (ScssExternalScanner) Destroy(payload any)

func (ScssExternalScanner) Scan 

func (ScssExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (ScssExternalScanner) Serialize 

func (ScssExternalScanner) Serialize(payload any, buf []byte) int

func (ScssExternalScanner) SupportsIncrementalReuse 

func (ScssExternalScanner) SupportsIncrementalReuse() bool

type SqlExternalScanner 

type SqlExternalScanner struct{}

SqlExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-sql.

This is a Go port of the C external scanner from DerekStride/tree-sitter-sql. The scanner handles PostgreSQL dollar-quoted strings: $tag$..content..$tag$. It uses state to remember the opening tag so the closing tag can be matched.

func (SqlExternalScanner) Create 

func (SqlExternalScanner) Create() any

func (SqlExternalScanner) Deserialize 

func (SqlExternalScanner) Deserialize(payload any, buf []byte)

func (SqlExternalScanner) Destroy 

func (SqlExternalScanner) Destroy(payload any)

func (SqlExternalScanner) Scan 

func (SqlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (SqlExternalScanner) Serialize 

func (SqlExternalScanner) Serialize(payload any, buf []byte) int

type SquirrelExternalScanner 

type SquirrelExternalScanner struct{}

SquirrelExternalScanner handles @"..." verbatim string literals for Squirrel. Inside a verbatim string, "" is an escaped double-quote.

func (SquirrelExternalScanner) Create 

func (SquirrelExternalScanner) Create() any

func (SquirrelExternalScanner) Deserialize 

func (SquirrelExternalScanner) Deserialize(payload any, buf []byte)

func (SquirrelExternalScanner) Destroy 

func (SquirrelExternalScanner) Destroy(payload any)

func (SquirrelExternalScanner) Scan 

func (SquirrelExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (SquirrelExternalScanner) Serialize 

func (SquirrelExternalScanner) Serialize(payload any, buf []byte) int

type StarlarkExternalScanner 

type StarlarkExternalScanner struct{}

StarlarkExternalScanner handles indent/dedent and string literals for Starlark. Starlark is essentially Python syntax; this reuses the pythonScannerState type.

func (StarlarkExternalScanner) Create 

func (StarlarkExternalScanner) Create() any

func (StarlarkExternalScanner) Deserialize 

func (StarlarkExternalScanner) Deserialize(payload any, buf []byte)

func (StarlarkExternalScanner) Destroy 

func (StarlarkExternalScanner) Destroy(payload any)

func (StarlarkExternalScanner) Scan 

func (StarlarkExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (StarlarkExternalScanner) Serialize 

func (StarlarkExternalScanner) Serialize(payload any, buf []byte) int

func (StarlarkExternalScanner) SupportsIncrementalReuse 

func (StarlarkExternalScanner) SupportsIncrementalReuse() bool

type SvelteExternalScanner 

type SvelteExternalScanner struct{}

SvelteExternalScanner handles HTML tag tracking plus Svelte-specific raw text scanning (for expression blocks like {#each}, {@html}, etc.) and special sigil characters (@, #, /, :).

func (SvelteExternalScanner) Create 

func (SvelteExternalScanner) Create() any

func (SvelteExternalScanner) Deserialize 

func (SvelteExternalScanner) Deserialize(payload any, buf []byte)

func (SvelteExternalScanner) Destroy 

func (SvelteExternalScanner) Destroy(any)

func (SvelteExternalScanner) Scan 

func (SvelteExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (SvelteExternalScanner) Serialize 

func (SvelteExternalScanner) Serialize(payload any, buf []byte) int

func (SvelteExternalScanner) SupportsIncrementalReuse 

func (SvelteExternalScanner) SupportsIncrementalReuse() bool

type SwiftExternalScanner 

type SwiftExternalScanner struct {
	// contains filtered or unexported fields
}

SwiftExternalScanner handles all external tokens for the Swift grammar.

func (SwiftExternalScanner) Create 

func (SwiftExternalScanner) Create() any

func (SwiftExternalScanner) Deserialize 

func (SwiftExternalScanner) Deserialize(payload any, buf []byte)

func (SwiftExternalScanner) Destroy 

func (SwiftExternalScanner) Destroy(payload any)

func (SwiftExternalScanner) ExternalScannerForLanguage 

func (SwiftExternalScanner) ExternalScannerForLanguage(lang *gotreesitter.Language) gotreesitter.ExternalScanner

func (SwiftExternalScanner) Scan 

func (s SwiftExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (SwiftExternalScanner) Serialize 

func (SwiftExternalScanner) Serialize(payload any, buf []byte) int

type TablegenExternalScanner 

type TablegenExternalScanner struct{}

TablegenExternalScanner handles nestable /* */ comments for TableGen.

func (TablegenExternalScanner) Create 

func (TablegenExternalScanner) Create() any

func (TablegenExternalScanner) Deserialize 

func (TablegenExternalScanner) Deserialize(payload any, buf []byte)

func (TablegenExternalScanner) Destroy 

func (TablegenExternalScanner) Destroy(payload any)

func (TablegenExternalScanner) Scan 

func (TablegenExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TablegenExternalScanner) Serialize 

func (TablegenExternalScanner) Serialize(payload any, buf []byte) int

type TclExternalScanner 

type TclExternalScanner struct{}

TclExternalScanner handles concatenation detection for Tcl.

func (TclExternalScanner) Create 

func (TclExternalScanner) Create() any

func (TclExternalScanner) Deserialize 

func (TclExternalScanner) Deserialize(payload any, buf []byte)

func (TclExternalScanner) Destroy 

func (TclExternalScanner) Destroy(payload any)

func (TclExternalScanner) Scan 

func (TclExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TclExternalScanner) Serialize 

func (TclExternalScanner) Serialize(payload any, buf []byte) int

type TealExternalScanner 

type TealExternalScanner struct{}

TealExternalScanner handles Teal/Lua string and comment scanning.

func (TealExternalScanner) Create 

func (TealExternalScanner) Create() any

func (TealExternalScanner) Deserialize 

func (TealExternalScanner) Deserialize(payload any, buf []byte)

func (TealExternalScanner) Destroy 

func (TealExternalScanner) Destroy(payload any)

func (TealExternalScanner) Scan 

func (TealExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TealExternalScanner) Serialize 

func (TealExternalScanner) Serialize(payload any, buf []byte) int

type TemplExternalScanner 

type TemplExternalScanner struct{}

TemplExternalScanner handles CSS property values, script blocks, and element text for templ.

func (TemplExternalScanner) Create 

func (TemplExternalScanner) Create() any

func (TemplExternalScanner) Deserialize 

func (TemplExternalScanner) Deserialize(payload any, buf []byte)

func (TemplExternalScanner) Destroy 

func (TemplExternalScanner) Destroy(payload any)

func (TemplExternalScanner) Scan 

func (TemplExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TemplExternalScanner) Serialize 

func (TemplExternalScanner) Serialize(payload any, buf []byte) int

type TlaplusExternalScanner 

type TlaplusExternalScanner struct{}

TlaplusExternalScanner handles junction lists, proofs, extramodular text, fairness operators, and PlusCal context nesting for the TLA+ grammar.

func (TlaplusExternalScanner) Create 

func (TlaplusExternalScanner) Create() any

func (TlaplusExternalScanner) Deserialize 

func (TlaplusExternalScanner) Deserialize(payload any, buf []byte)

func (TlaplusExternalScanner) Destroy 

func (TlaplusExternalScanner) Destroy(payload any)

func (TlaplusExternalScanner) Scan 

func (TlaplusExternalScanner) Scan(
	payload any,
	lexer *gotreesitter.ExternalLexer,
	validSymbols []bool,
) bool

func (TlaplusExternalScanner) Serialize 

func (TlaplusExternalScanner) Serialize(payload any, buf []byte) int

type TomlExternalScanner 

type TomlExternalScanner struct{}

TomlExternalScanner ports tree-sitter-toml's stateless external scanner.

func (TomlExternalScanner) Create 

func (TomlExternalScanner) Create() any

func (TomlExternalScanner) Deserialize 

func (TomlExternalScanner) Deserialize(payload any, buf []byte)

func (TomlExternalScanner) Destroy 

func (TomlExternalScanner) Destroy(payload any)

func (TomlExternalScanner) Scan 

func (TomlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TomlExternalScanner) Serialize 

func (TomlExternalScanner) Serialize(payload any, buf []byte) int

func (TomlExternalScanner) SupportsIncrementalReuse 

func (TomlExternalScanner) SupportsIncrementalReuse() bool

type TomlTokenSource 

type TomlTokenSource struct {
	// contains filtered or unexported fields
}

TomlTokenSource is a lightweight lexer bridge for tree-sitter-toml. It focuses on practical coverage for common editor workflows and incremental parsing.

func NewTomlTokenSource 

func NewTomlTokenSource(src []byte, lang *gotreesitter.Language) (*TomlTokenSource, error)

NewTomlTokenSource creates a token source for TOML source text.

func (*TomlTokenSource) Next 

func (ts *TomlTokenSource) Next() gotreesitter.Token

func (*TomlTokenSource) Reset 

func (ts *TomlTokenSource) Reset(src []byte)

Reset reinitializes this token source for a new source buffer.

func (*TomlTokenSource) SkipToByte 

func (ts *TomlTokenSource) SkipToByte(offset uint32) gotreesitter.Token

func (*TomlTokenSource) SupportsIncrementalReuse 

func (ts *TomlTokenSource) SupportsIncrementalReuse() bool

SupportsIncrementalReuse reports that TomlTokenSource preserves stable token boundaries across edits and supports deterministic SkipToByte behavior.

type TsxExternalScanner 

type TsxExternalScanner struct{}

TsxExternalScanner handles automatic semicolons, template strings, JSX text, ternary question marks, and HTML comments for TSX.

func (TsxExternalScanner) Create 

func (TsxExternalScanner) Create() any

func (TsxExternalScanner) Deserialize 

func (TsxExternalScanner) Deserialize(payload any, buf []byte)

func (TsxExternalScanner) Destroy 

func (TsxExternalScanner) Destroy(payload any)

func (TsxExternalScanner) Scan 

func (TsxExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TsxExternalScanner) Serialize 

func (TsxExternalScanner) Serialize(payload any, buf []byte) int

func (TsxExternalScanner) SupportsIncrementalReuse 

func (TsxExternalScanner) SupportsIncrementalReuse() bool

type TypeScriptExternalScanner 

type TypeScriptExternalScanner struct{}

TypeScriptExternalScanner handles automatic semicolons, template strings, JSX text, ternary question marks, and HTML comments for TypeScript.

func (TypeScriptExternalScanner) Create 

func (TypeScriptExternalScanner) Create() any

func (TypeScriptExternalScanner) Deserialize 

func (TypeScriptExternalScanner) Deserialize(payload any, buf []byte)

func (TypeScriptExternalScanner) Destroy 

func (TypeScriptExternalScanner) Destroy(payload any)

func (TypeScriptExternalScanner) Scan 

func (TypeScriptExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TypeScriptExternalScanner) Serialize 

func (TypeScriptExternalScanner) Serialize(payload any, buf []byte) int

func (TypeScriptExternalScanner) SupportsIncrementalReuse 

func (TypeScriptExternalScanner) SupportsIncrementalReuse() bool

type TypstExternalScanner 

type TypstExternalScanner struct{}

TypstExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-typst.

This is a Go port of the C external scanner from tree-sitter-typst (https://github.com/Enter-tainer/tree-sitter-typst). The scanner handles 49 external tokens including indentation tracking, content/strong/emph containers, raw blocks, comments, headings, list items, math mode, and more.

func (TypstExternalScanner) Create 

func (TypstExternalScanner) Create() any

func (TypstExternalScanner) Deserialize 

func (TypstExternalScanner) Deserialize(payload any, buf []byte)

func (TypstExternalScanner) Destroy 

func (TypstExternalScanner) Destroy(payload any)

func (TypstExternalScanner) Scan 

func (TypstExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (TypstExternalScanner) Serialize 

func (TypstExternalScanner) Serialize(payload any, buf []byte) int

type UxntalExternalScanner 

type UxntalExternalScanner struct{}

UxntalExternalScanner handles nestable ( ) Forth-style comments for Uxntal.

func (UxntalExternalScanner) Create 

func (UxntalExternalScanner) Create() any

func (UxntalExternalScanner) Deserialize 

func (UxntalExternalScanner) Deserialize(payload any, buf []byte)

func (UxntalExternalScanner) Destroy 

func (UxntalExternalScanner) Destroy(payload any)

func (UxntalExternalScanner) Scan 

func (UxntalExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, _ []bool) bool

func (UxntalExternalScanner) Serialize 

func (UxntalExternalScanner) Serialize(payload any, buf []byte) int

type VhdlExternalScanner 

type VhdlExternalScanner struct{}

VhdlExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-vhdl (jpt13653903). It handles identifier/keyword matching, number/string literals, comments, directives, and library tokens.

func (VhdlExternalScanner) Create 

func (VhdlExternalScanner) Create() any

func (VhdlExternalScanner) Deserialize 

func (VhdlExternalScanner) Deserialize(payload any, buf []byte)

func (VhdlExternalScanner) Destroy 

func (VhdlExternalScanner) Destroy(payload any)

func (VhdlExternalScanner) Scan 

func (VhdlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (VhdlExternalScanner) Serialize 

func (VhdlExternalScanner) Serialize(payload any, buf []byte) int

type VueExternalScanner 

type VueExternalScanner struct{}

VueExternalScanner handles HTML tag tracking plus Vue-specific text fragments and interpolation.

func (VueExternalScanner) Create 

func (VueExternalScanner) Create() any

func (VueExternalScanner) Deserialize 

func (VueExternalScanner) Deserialize(payload any, buf []byte)

func (VueExternalScanner) Destroy 

func (VueExternalScanner) Destroy(payload any)

func (VueExternalScanner) Scan 

func (VueExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (VueExternalScanner) Serialize 

func (VueExternalScanner) Serialize(payload any, buf []byte) int

type WalkStats 

type WalkStats struct {
	FilesFound    int   // files with a recognized language extension
	FilesParsed   int   // files successfully parsed into a BoundTree
	FilesFailed   int   // files that encountered read or parse errors
	FilesFiltered int   // files skipped by SkipExtensions or ShouldParse
	LargeFiles    int   // files at or above LargeFileThreshold
	BinarySkipped int   // files detected as binary (NUL in first 8 KB)
	BytesParsed   int64 // total bytes of successfully parsed files
}

WalkStats summarizes the results of a WalkAndParse run. The stats function returned by WalkAndParse blocks until all work is done, then returns this snapshot.

type WgslExternalScanner 

type WgslExternalScanner struct{}

WgslExternalScanner handles nestable /* */ block comments for WGSL.

func (WgslExternalScanner) Create 

func (WgslExternalScanner) Create() any

func (WgslExternalScanner) Deserialize 

func (WgslExternalScanner) Deserialize(payload any, buf []byte)

func (WgslExternalScanner) Destroy 

func (WgslExternalScanner) Destroy(payload any)

func (WgslExternalScanner) Scan 

func (WgslExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (WgslExternalScanner) Serialize 

func (WgslExternalScanner) Serialize(payload any, buf []byte) int

type WolframExternalScanner 

type WolframExternalScanner struct{}

WolframExternalScanner handles nestable (* *) comments for Wolfram/Mathematica.

func (WolframExternalScanner) Create 

func (WolframExternalScanner) Create() any

func (WolframExternalScanner) Deserialize 

func (WolframExternalScanner) Deserialize(payload any, buf []byte)

func (WolframExternalScanner) Destroy 

func (WolframExternalScanner) Destroy(payload any)

func (WolframExternalScanner) Scan 

func (WolframExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (WolframExternalScanner) Serialize 

func (WolframExternalScanner) Serialize(payload any, buf []byte) int

type XMLExternalScanner 

type XMLExternalScanner struct{}

XMLExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-xml.

This is a Go port of the C external scanner from tree-sitter-xml (https://github.com/tree-sitter-grammars/tree-sitter-xml). The scanner manages a tag name stack and handles 11 external tokens: PITarget, PIContent, Comment, CharData, CData, xml-model, xml-stylesheet, StartTagName, EndTagName, ErroneousEndName, and SelfClosingTagDelimiter.

func (XMLExternalScanner) Create 

func (XMLExternalScanner) Create() any

func (XMLExternalScanner) Deserialize 

func (XMLExternalScanner) Deserialize(payload any, buf []byte)

func (XMLExternalScanner) Destroy 

func (XMLExternalScanner) Destroy(payload any)

func (XMLExternalScanner) Scan 

func (XMLExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (XMLExternalScanner) Serialize 

func (XMLExternalScanner) Serialize(payload any, buf []byte) int

type YamlExternalScanner 

type YamlExternalScanner struct{}

func (YamlExternalScanner) Create 

func (YamlExternalScanner) Create() any

func (YamlExternalScanner) Deserialize 

func (YamlExternalScanner) Deserialize(payload any, buf []byte)

func (YamlExternalScanner) Destroy 

func (YamlExternalScanner) Destroy(payload any)

func (YamlExternalScanner) Scan 

func (YamlExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (YamlExternalScanner) Serialize 

func (YamlExternalScanner) Serialize(payload any, buf []byte) int

type YuckExternalScanner 

type YuckExternalScanner struct{}

YuckExternalScanner implements gotreesitter.ExternalScanner for tree-sitter-yuck. Handles unescaped string fragments for single-quote, double-quote, and backtick strings.

func (YuckExternalScanner) Create 

func (YuckExternalScanner) Create() any

func (YuckExternalScanner) Deserialize 

func (YuckExternalScanner) Deserialize(payload any, buf []byte)

func (YuckExternalScanner) Destroy 

func (YuckExternalScanner) Destroy(payload any)

func (YuckExternalScanner) Scan 

func (YuckExternalScanner) Scan(payload any, lexer *gotreesitter.ExternalLexer, validSymbols []bool) bool

func (YuckExternalScanner) Serialize 

func (YuckExternalScanner) Serialize(payload any, buf []byte) int

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.