textmate

README

textmate-go

textmate-go is a pure-Go port of Microsoft's vscode-textmate. It provides line-oriented TextMate grammar tokenization, reusable state stacks, and incremental document highlighting without CGO.

• Exact token-text and scope-stack parity is continuously checked against vscode-textmate.
• The optional grammars package provides 124 bundled languages.
• Immutable states and interned scope stacks support efficient editor caches.
• Size and time limits bound work on untrusted or minified input.
• Go 1.25 or newer is required. CGO_ENABLED=0 is supported.

The project is under active development, so its API may evolve before a stable release.

Installation

go get github.com/eugenioenko/textmate-go@latest

Quick start

package main

import (
	"log"

	textmate "github.com/eugenioenko/textmate-go"
	"github.com/eugenioenko/textmate-go/grammars"
)

func main() {
	registry := textmate.NewRegistry(textmate.RegistryOptions{
		LoadGrammar: grammars.Load,
	})
	defer registry.Dispose()

	grammar, err := registry.LoadGrammar(grammars.ScopeForFilename("main.go"))
	if err != nil {
		log.Fatal(err)
	}
	if grammar == nil {
		log.Fatal("no embedded grammar for main.go")
	}

	var state *textmate.StateStack
	for _, line := range []string{"package main", `const message = "hello"`} {
		result := grammar.TokenizeLine(line, state)
		for _, token := range result.Tokens {
			log.Printf("%d:%d %v", token.Start, token.End, token.Scopes)
		}
		state = result.RuleStack
	}
}

Token offsets are rune offsets. Scopes are ordered from the root grammar to the most specific matched scope. Pass nil or textmate.InitialState for the first line, then carry RuleStack to the next line.

See the usage guide for editable documents, state identity, interned scope stacks, token categories, and tokenization limits.

Embedded grammars

The optional grammars package embeds 124 license-reviewed grammar roots plus one MIT support grammar used for Markdown's inline HTML. Assets are individually compressed and loaded once on demand.

• A-D: ActionScript, AutoHotkey, Angular HTML, Assembly, AWK, Ballerina, Batch File, BibTeX, Bicep, C, C3, Chapel, Clojure, CMake, COBOL, CoffeeScript, Common Lisp, Rocq, C++, Crystal, C#, CSS, CSV, D, Dart, Desktop, Diff, Dockerfile, and dotEnv.
• E-H: Elixir, Elm, Emacs Lisp, Erlang, Fennel, Fish, F#, GDScript, Gherkin, Gleam, Go, GraphQL, Groovy, Handlebars, Haskell, Haxe, HashiCorp HCL, HLSL, HTML, HTTP, and Hy.
• I-M: INI, Java, JavaScript, Jinja, JSON, JSON with Comments, Jsonnet, JSX, Julia, KDL, Kotlin, Lean 4, Less, Lua, Makefile, Markdown, Mojo, and MoonBit.
• N-R: Nix, nushell, Objective-C, Objective-C++, OCaml, Odin, OpenSCAD, Pascal, Perl, PHP, PL/SQL, PowerQuery, PowerShell, Protocol Buffer 3, Puppet, Python, QML, R, Windows Registry Script, reStructuredText, Ruby, and Rust.
• S-Z: SAS, Sass, Scala, Scheme, SCSS, Shell, Shell Session, GNU Smalltalk, Solidity, SQL, Stylus, Svelte, Swift, SystemVerilog, Systemd Units, Terraform, TeX, TOML, TSX, Twig, TypeScript, Typst, V, Vala, Visual Basic, Verilog, VHDL, Vim Script, Vue, WebAssembly, WGSL, XML, YAML, and Zig.

The internal html-derivative grammar supporting Markdown is not counted as a separate language. Canonical IDs, aliases, filename mappings, scopes, source revisions, and licenses are recorded in grammars/MANIFEST.md. Regeneration instructions and all-versus-curated size measurements are in grammars/README.md.

Regex compatibility and diagnostics

TextMate grammars require lookbehind, backreferences, and position-sensitive anchors that Go's standard RE2-based regexp package cannot provide. This port uses the maintained github.com/eugenioenko/regexp2/v2 fork without RE2 compatibility mode. The fork adds a text-free capture-index API and TextMate-oriented search optimizations.

Oniguruma-only syntax is translated before compilation. Unsupported constructs, compile failures, match panics, and timeouts are non-fatal and available through grammar.Diagnostics() so one bad pattern does not stop the tokenizer.

The fork has its own module path, so downstream users receive it as a normal transitive dependency without a replace directive.

Verification

The GitHub Actions merge gate runs on pull requests, main, version tags, and manual dispatches. A green required check covers:

• formatting, vet, lint, generated-file cleanliness, pure-Go builds, unit tests, race tests, and a bounded fuzz run;
• all 101 upstream tokenization tests against vscode-textmate and textmate-go;
• exact differential tokenization across the real-file corpus and deterministic edit, splice, and UTF-8 fuzz cases;
• upstream theme tests, 72 theme scope differentials, and 72 isolated golden fixtures across 14 themes;
• standalone tokenization of all 166 embedded roots, plus parsing and loading all 260 pinned source grammars and scanning their 34,698 regex fields;
• base-versus-PR performance results for all 20 benchmark cases, posted as one updated PR comment.

Benchmark timing is informational because shared runners are noisy. Correctness, corpus, and benchmark execution failures block the merge.

See conformance/README.md for commands and oracle behavior, grammars/README.md for updating the grammar report golden file, and benchmarks/README.md for local benchmark reproduction and CI comment behavior.

Performance

Warm line-by-line tokenization from make bench-compare-extended, in microseconds per line, using the fastest of five Linux/amd64 runs with Go 1.25.1 on an AMD Ryzen 7 6800H. The textmate-go column was measured on 2026-10-06 with regexp2 v2.8.2; the vscode-textmate and Chroma columns are from 2026-09-25 on the same machine. Ratios below 1 mean textmate-go is faster.

Chroma starts each line from its root state, so its column is a throughput reference rather than an equivalent incremental result.

Case	textmate-go	allocs/line	vscode-textmate	Chroma	vs JS	vs Chroma
TSX	19.8	24.0	20.8	17.1	0.95x	1.16x
HTML	19.7	37.4	26.0	5.2	0.76x	3.78x
Go	8.0	17.9	14.0	20.3	0.57x	0.39x
Markdown	15.0	19.3	15.3	19.1	0.98x	0.79x
TypeScript	60.1	36.6	58.2	26.3	1.03x	2.29x
JavaScript	52.2	36.2	64.3	26.7	0.81x	1.95x
CSS	23.3	31.9	74.0	11.2	0.31x	2.08x
JSON	7.9	29.4	7.6	9.3	1.04x	0.85x
Python	34.1	34.2	48.4	53.6	0.70x	0.64x
Rust	17.2	26.5	31.2	29.7	0.55x	0.58x
Java	42.0	30.8	38.9	35.5	1.08x	1.18x
C++	164.1	67.8	158.0	45.0	1.04x	3.65x
Ruby	25.2	25.9	59.0	62.6	0.43x	0.40x
Shell	25.1	42.7	31.8	23.0	0.79x	1.09x
C#	24.3	24.7	24.9	14.1	0.98x	1.73x
PHP	13.1	18.6	24.6	10.9	0.53x	1.20x
SQL	11.8	15.2	41.9	25.5	0.28x	0.46x
YAML	8.2	17.8	11.2	7.7	0.73x	1.07x
Haskell	6.3	15.2	9.0	7.2	0.70x	0.88x
Vue	21.0	31.2	28.6	14.9	0.74x	1.41x

textmate-go is faster than vscode-textmate in sixteen of the twenty cases and within 8% in the other four: TypeScript, JSON, Java, and C++. See benchmarks/README.md for the method and corpus, and docs/performance-baseline.md for the earlier Phase 4 baseline.

Development

make test
make bench
make bench-compare
make bench-compare-extended
make lint
CGO_ENABLED=0 go build ./...

make bench-compare needs a compiled vscode-textmate checkout and the conformance harness dependencies. See benchmarks/README.md for setup and options.

Pinned revisions used by development and conformance testing:

• vscode-textmate: fbe49961ab8077e587fdf5282019655ae69e5f9e
• textmate-grammars-themes: 37edd1b26f18838050661d912334aba0ca7f4931
• gopher-textmate: ce2b42e5386c93ae781add9df2e6328338b06f9e
• regexp2: maintained fork github.com/eugenioenko/regexp2/v2, release v2.8.1, based on upstream v2.8.0 commit 9d0d2ffe88a8b90012f7979ec85424e46d5ef48f

See plan.md for implementation phases and conformance goals.

Acknowledgments and thanks

This project would not exist without Microsoft's vscode-textmate, whose implementation defines the behavior that this Go port follows, and Shiki's textmate-grammars-themes, which provides the pinned grammar corpus, source provenance, and license metadata used to build the optional embedded grammar package. Thanks also to the language-extension and grammar authors whose work is collected there.

The tokenizer relies on the work behind regexp2 for the regular-expression features TextMate grammars require. Microsoft's vscode-oniguruma provides an essential compatibility oracle. We are grateful to all of these projects and their contributors.

The Microsoft MIT license for vscode-textmate is reproduced in THIRD_PARTY_NOTICES.

License

MIT. See LICENSE.

Documentation

Overview

Package textmate provides pure-Go, line-oriented tokenization using TextMate grammars. Token offsets are rune indices. Callers pass each LineResult's immutable RuleStack into the next line and may compare saved stacks with StateStack.Equal when incrementally re-tokenizing edited documents.

Index

• Constants
• Variables
• type Document
  • func NewDocument(grammar *Grammar, options DocumentOptions) *Document
  • func (d *Document) InvalidateFrom(index int) error
  • func (d *Document) Len() int
  • func (d *Document) Line(index int) (LineResult, bool)
  • func (d *Document) MaterializedLines() int
  • func (d *Document) ReplaceLines(start, end int, replacement []string) error
  • func (d *Document) SetLines(lines []string)
  • func (d *Document) StateAt(index int) (*StateStack, bool)
• type DocumentOptions
• type Grammar
  • func (g *Grammar) Diagnostics() []oniguruma.Diagnostic
  • func (g *Grammar) TokenizeLine(line string, prev *StateStack) LineResult
  • func (g *Grammar) TokenizeLineWithOptions(line string, prev *StateStack, options TokenizeOptions) LineResult
• type IncludeString
• type LineResult
• type Location
• type RawCaptures
  • func (captures *RawCaptures) UnmarshalJSON(data []byte) error
• type RawGrammar
  • func ParseRawGrammar(data []byte) (*RawGrammar, error)
• type RawInjections
• type RawRepository
  • func (repository *RawRepository) UnmarshalJSON(data []byte) error
• type RawRule
  • func (r *RawRule) UnmarshalJSON(data []byte) error
• type RegExpString
• type Registry
  • func NewRegistry(options RegistryOptions) *Registry
  • func (r *Registry) Dispose()
  • func (r *Registry) LoadGrammar(scopeName string) (*Grammar, error)
• type RegistryOptions
• type RuleFactory
• type ScopeStack
  • func (s *ScopeStack) At(index int) (string, bool)
  • func (s *ScopeStack) Category() TokenCategory
  • func (s *ScopeStack) ID() ScopeStackID
  • func (s *ScopeStack) Len() int
  • func (s *ScopeStack) Names() []string
  • func (s *ScopeStack) Range(yield func(string) bool)
• type ScopeStackID
  • func (id ScopeStackID) IsZero() bool
• type StateStack
  • func (s *StateStack) Equal(other *StateStack) bool
  • func (s *StateStack) String() string
• type StopReason
  • func (r StopReason) String() string
• type Token
  • func (token Token) Category() TokenCategory
• type TokenCategory
  • func ClassifyScopes(scopes []string) TokenCategory
  • func (category TokenCategory) String() string
• type TokenizeOptions

Constants

const DefaultDocumentCacheCapacity = 20_000

DefaultDocumentCacheCapacity is the number of distinct line/state results retained by a Document when DocumentOptions.CacheCapacity is zero.

Variables

var ErrInvalidLineRange = errors.New("textmate: invalid document line range")

ErrInvalidLineRange is returned when ReplaceLines receives a range outside the current document or one whose start is after its end.

var InitialState = newStateStack(nil, 0, 0, 0, false, nil, nil, nil)

InitialState is the sentinel passed when tokenizing the first line.

Types

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document incrementally tokenizes an editable sequence of lines. It owns a shallow copy of the supplied line slice, carries start states between lines, and reuses an unchanged suffix after an edit as soon as tokenizer state converges with the previous suffix.

All methods are safe to call concurrently. Results returned by Line are immutable, library-owned views and must not be modified; this permits cache hits without allocating. A result remains valid after later calls and cache eviction. Copy the Tokens slice before changing Token fields. ScopeStack and Token.Scopes remain library-owned and read-only, as they are for Grammar.TokenizeLine.

func NewDocument

func NewDocument(grammar *Grammar, options DocumentOptions) *Document

NewDocument creates an empty document backed by grammar. A nil grammar is valid: lines remain addressable, but Line returns empty tokenization results.

func (*Document) InvalidateFrom

func (d *Document) InvalidateFrom(index int) error

InvalidateFrom discards materialized states for line index and all following lines, and clears the result cache. Index may equal Len. The next Line or StateAt call recomputes the discarded range.

Time-limit stops are never cached, and a state that came from one is recomputed on the next Line, StateAt or MaterializedLines call, a bounded number of times. InvalidateFrom forces the complete tail to be recomputed regardless, for example after a grammar or theme change on the caller side.

func (*Document) Len

func (d *Document) Len() int

Len returns the current number of lines.

func (*Document) Line

func (d *Document) Line(index int) (LineResult, bool)

Line returns the tokenization result for line index. The boolean is false when index is outside the current document. Requesting a line computes any missing start states above it and records the line's resulting state.

The returned result and its Tokens slice are immutable, library-owned views. They remain valid after later calls and cache eviction. See Document's ownership documentation.

func (*Document) MaterializedLines

func (d *Document) MaterializedLines() int

MaterializedLines returns how many leading lines have a known start state. Line(index) for an index below it tokenizes only that line; a larger index first tokenizes every line from MaterializedLines up to it. An edit lowers it to the first changed line until recomputation converges with the old tail.

func (*Document) ReplaceLines

func (d *Document) ReplaceLines(start, end int, replacement []string) error

ReplaceLines replaces the half-open line range [start, end) with a shallow copy of replacement. It returns ErrInvalidLineRange without changing the document when the range is invalid.

Lines and states are spliced in place, so an edit costs the replaced lines plus shifting the ones after them, rather than copying and comparing the whole document as SetLines does.

func (*Document) SetLines

func (d *Document) SetLines(lines []string)

SetLines replaces the complete document with a shallow copy of lines. Tokenizer states above the common prefix are retained. States in the common suffix become reusable once recomputation reaches an equal canonical state.

func (*Document) StateAt

func (d *Document) StateAt(index int) (*StateStack, bool)

StateAt returns the state in which line index starts. Index may equal Len, in which case StateAt returns the state immediately after the final line. The boolean is false for all other out-of-range indices.

type DocumentOptions

type DocumentOptions struct {
	// TokenizeOptions is applied to every line tokenized by the document.
	TokenizeOptions TokenizeOptions

	// CacheCapacity bounds cached results keyed by line text and canonical
	// start state. Zero selects DefaultDocumentCacheCapacity. A negative value
	// disables result caching.
	CacheCapacity int
}

DocumentOptions configures a Document. Options are fixed at construction.

type Grammar

type Grammar struct {
	// contains filtered or unexported fields
}

Grammar is a compiled TextMate grammar. Compilation remains lazy: rules and their regular expressions are created only when the grammar is first used. A Grammar serializes tokenization because compiled rules contain mutable scanner caches.

func (*Grammar) Diagnostics

func (g *Grammar) Diagnostics() []oniguruma.Diagnostic

Diagnostics returns regex translation, compilation, and match diagnostics from rules compiled or exercised so far. Grammars compile lazily, so callers should take another snapshot after tokenizing new syntax paths. Equivalent diagnostics produced by anchor variants or shared rule lists are returned once.

func (*Grammar) TokenizeLine

func (g *Grammar) TokenizeLine(line string, prev *StateStack) LineResult

TokenizeLine tokenizes line and returns immutable state to pass to the next call. A nil state and InitialState both begin a new document.

func (*Grammar) TokenizeLineWithOptions

func (g *Grammar) TokenizeLineWithOptions(
	line string,
	prev *StateStack,
	options TokenizeOptions,
) LineResult

TokenizeLineWithOptions tokenizes line with optional soft time and line-size limits. StoppedAt is the first unparsed rune offset when Stopped is true. See TokenizeOptions for the precise limit semantics.

type IncludeString

type IncludeString = string

IncludeString is a TextMate include reference such as "#rule", "$self", or "source.js#rule".

type LineResult

type LineResult struct {
	Tokens    []Token
	RuleStack *StateStack
	// Stopped remains the compatibility signal for an incomplete parse.
	Stopped bool
	// StoppedReason is StopReasonNone when Stopped is false.
	StoppedReason StopReason
	// StoppedAt is the first unparsed rune offset when Stopped is true.
	StoppedAt int
}

LineResult is the result of tokenizing one line.

type Location

type Location struct {
	Filename string `json:"filename"`
	Line     int    `json:"line"`
	Char     int    `json:"char"`
}

Location identifies a value in a source grammar. Plain JSON grammars do not normally contain it, but vscode-textmate can attach it while parsing.

type RawCaptures

type RawCaptures map[string]*RawRule

RawCaptures maps numeric capture group names to their rules.

func (*RawCaptures) UnmarshalJSON

func (captures *RawCaptures) UnmarshalJSON(data []byte) error

UnmarshalJSON accepts both the specified object form and the array form emitted by a few converted TextMate grammars. Invalid capture entries are ignored, matching JavaScript's permissive property access on those values.

type RawGrammar

type RawGrammar struct {
	Repository        RawRepository `json:"repository"`
	ScopeName         string        `json:"scopeName"`
	Patterns          []*RawRule    `json:"patterns"`
	Injections        RawInjections `json:"injections,omitempty"`
	InjectionSelector string        `json:"injectionSelector,omitempty"`
	FileTypes         []string      `json:"fileTypes,omitempty"`
	Name              string        `json:"name,omitempty"`
	FirstLineMatch    RegExpString  `json:"firstLineMatch,omitempty"`
	Location          *Location     `json:"$vscodeTextmateLocation,omitempty"`
}

RawGrammar is the JSON representation of a TextMate grammar.

func ParseRawGrammar

func ParseRawGrammar(data []byte) (*RawGrammar, error)

ParseRawGrammar decodes a JSON TextMate grammar.

type RawInjections

type RawInjections map[string]*RawRule

RawInjections maps scope selector expressions to injected rules.

type RawRepository

type RawRepository map[string]*RawRule

RawRepository maps grammar-defined repository names to rules.

func (*RawRepository) UnmarshalJSON

func (repository *RawRepository) UnmarshalJSON(data []byte) error

UnmarshalJSON ignores malformed non-object entries found in a small number of converted grammars. JavaScript loads these values but they do not compile into meaningful rules.

type RawRule

type RawRule struct {
	Include *IncludeString `json:"include,omitempty"`

	Name        *string `json:"name,omitempty"`
	ContentName *string `json:"contentName,omitempty"`

	Match         *RegExpString `json:"match,omitempty"`
	Captures      RawCaptures   `json:"captures,omitempty"`
	Begin         *RegExpString `json:"begin,omitempty"`
	BeginCaptures RawCaptures   `json:"beginCaptures,omitempty"`
	End           *RegExpString `json:"end,omitempty"`
	EndCaptures   RawCaptures   `json:"endCaptures,omitempty"`
	While         *RegExpString `json:"while,omitempty"`
	WhileCaptures RawCaptures   `json:"whileCaptures,omitempty"`
	Patterns      []*RawRule    `json:"patterns,omitempty"`

	Repository          RawRepository `json:"repository,omitempty"`
	ApplyEndPatternLast bool          `json:"applyEndPatternLast,omitempty"`
	Location            *Location     `json:"$vscodeTextmateLocation,omitempty"`
}

RawRule is one rule in a TextMate grammar.

func (*RawRule) UnmarshalJSON

func (r *RawRule) UnmarshalJSON(data []byte) error

type RegExpString

type RegExpString = string

RegExpString is a regular expression in the grammar's Oniguruma dialect.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry owns raw and compiled grammars and resolves their transitive dependencies. Callback invocations never occur while the registry lock is held.

func NewRegistry

func NewRegistry(options RegistryOptions) *Registry

NewRegistry constructs an empty lazy-loading grammar registry.

func (*Registry) Dispose

func (r *Registry) Dispose()

Dispose releases the registry's references to loaded grammars. It is idempotent; subsequent loads fail.

func (*Registry) LoadGrammar

func (r *Registry) LoadGrammar(scopeName string) (*Grammar, error)

LoadGrammar loads scopeName and all reachable includes and injections, then returns the cached compiled grammar for scopeName.

type RegistryOptions

type RegistryOptions struct {
	LoadGrammar   func(scopeName string) (*RawGrammar, error)
	GetInjections func(scopeName string) []string
}

RegistryOptions provides the callbacks used to lazily resolve grammars. A loader may return (nil, nil) when a scope is not available. Missing root grammars are errors; missing optional dependencies are ignored.

type RuleFactory

type RuleFactory struct {
	// contains filtered or unexported fields
}

RuleFactory turns raw grammar nodes into registered rules. A factory belongs to one compiled Grammar; keeping raw node identity here avoids mutating the public JSON model with compilation state.

type ScopeStack

type ScopeStack struct {
	// contains filtered or unexported fields
}

ScopeStack is an immutable, interned sequence of TextMate scope names. Equal scope sequences produced by one Grammar have the same *ScopeStack and ScopeStackID, including when the tokenizer reconstructed its internal stack. A ScopeStack remains valid after its Grammar or Registry is no longer used.

ScopeStack values are owned by the library and must not be modified. Its accessors are safe to call concurrently.

func (*ScopeStack) At

func (s *ScopeStack) At(index int) (string, bool)

At returns the scope name at index and whether index is in range.

func (*ScopeStack) Category

func (s *ScopeStack) Category() TokenCategory

Category returns the cached coarse category of s. A nil ScopeStack is plain. The first call computes the result; subsequent calls are lock-free and allocate no memory.

func (*ScopeStack) ID

func (s *ScopeStack) ID() ScopeStackID

ID returns the stable runtime identity of s. A nil ScopeStack has the zero ID.

func (*ScopeStack) Len

func (s *ScopeStack) Len() int

Len returns the number of scope names in s.

func (*ScopeStack) Names

func (s *ScopeStack) Names() []string

Names returns a fresh copy of the scope names, ordered outermost first.

func (*ScopeStack) Range

func (s *ScopeStack) Range(yield func(string) bool)

Range calls yield for each scope name, ordered outermost first. Iteration stops when yield returns false.

type ScopeStackID

type ScopeStackID struct {
	// contains filtered or unexported fields
}

ScopeStackID identifies one interned scope-name stack. IDs are comparable and unique within the current process, so they can be used directly as map keys. The zero value does not identify a scope stack.

An ID remains valid for as long as the ScopeStack that returned it is reachable. IDs are runtime identities and must not be persisted.

func (ScopeStackID) IsZero

func (id ScopeStackID) IsZero() bool

IsZero reports whether id is the zero value.

type StateStack

type StateStack struct {
	// contains filtered or unexported fields
}

StateStack is an immutable tokenizer state. A returned state can safely be retained and compared after subsequent lines have been tokenized.

While two equal states returned by the same Grammar are both live, they are represented by the same *StateStack. This makes pointers suitable as short-lived cache keys. The identity is scoped to one Grammar and lasts only as long as the state remains reachable; use Equal after discarding all previously returned equal states. States are not transferable between grammars, and no pointer identity is promised across Grammar instances.

func (*StateStack) Equal

func (s *StateStack) Equal(other *StateStack) bool

Equal reports whether two stacks represent the same reusable tokenizer state. Line-local enter and anchor positions are intentionally ignored, as they are by vscode-textmate.

func (*StateStack) String

func (s *StateStack) String() string

type StopReason

type StopReason uint8

StopReason explains why a LineResult stopped before fully parsing its line.

const (
	// StopReasonNone means the line was fully tokenized.
	StopReasonNone StopReason = iota
	// StopReasonTimeLimit means the soft time budget was exhausted.
	StopReasonTimeLimit
	// StopReasonLineLimit means a byte or rune line-length cap was exceeded.
	StopReasonLineLimit
)

func (StopReason) String

func (r StopReason) String() string

type Token

type Token struct {
	Start      int
	End        int
	Scopes     []string
	ScopeStack *ScopeStack `json:"-"`
}

Token is one contiguous span of a tokenized line. Start and End are rune offsets into the original line, and Scopes is ordered outermost first.

ScopeStack is the immutable, comparable identity for Scopes. Equal scope sequences from the same Grammar reuse both ScopeStack and the backing array of Scopes. Scopes is retained for source compatibility and must be treated as read-only; callers that need to modify it should copy it first.

func (Token) Category

func (token Token) Category() TokenCategory

Category returns the token's coarse semantic category. It uses the cached ScopeStack classification when available and falls back to Scopes for tokens constructed by callers or older integrations.

type TokenCategory

type TokenCategory uint8

TokenCategory is a coarse semantic classification derived from TextMate scope names. It is intentionally independent of colors and font styles. Applications can map categories to their own presentation or use a full TextMate theme resolver when selector-accurate styling is required.

const (
	TokenCategoryPlain TokenCategory = iota
	TokenCategoryComment
	TokenCategoryString
	TokenCategoryRegexp
	TokenCategoryNumber
	TokenCategoryKeyword
	TokenCategoryOperator
	TokenCategoryFunction
	TokenCategoryTag
	TokenCategoryAttribute
	TokenCategoryType
	TokenCategoryBuiltin
	TokenCategoryVariable
	TokenCategoryHeading
	TokenCategoryBold
	TokenCategoryItalic
	TokenCategoryQuote
	TokenCategoryInserted
	TokenCategoryDeleted
	TokenCategoryPunctuation
	TokenCategoryInvalid
)

func ClassifyScopes

func ClassifyScopes(scopes []string) TokenCategory

ClassifyScopes returns a coarse category for a TextMate scope stack ordered outermost first. The innermost recognized scope wins. Within one scope, the longest prefix at a dot boundary wins. Plain is returned when no scope is recognized and for deliberate reset scopes such as meta.embedded.

Classification is a heuristic semantic layer, not TextMate theme resolution. The function does not retain or modify scopes and allocates no memory.

func (TokenCategory) String

func (category TokenCategory) String() string

String returns the stable, lower-case name of category.

type TokenizeOptions

type TokenizeOptions struct {
	TimeLimit    time.Duration
	MaxLineBytes int
	MaxLineRunes int
}

TokenizeOptions bounds work performed by TokenizeLineWithOptions. Zero or negative values disable the corresponding limit.

TimeLimit is a soft budget for the complete call, including waiting for the grammar lock, lazy root compilation, and begin/while condition setup. The tokenizer checks the budget between regexp searches; an in-flight regexp search cannot be interrupted and can therefore overrun the budget.

MaxLineBytes and MaxLineRunes are independent. A line is skipped when it exceeds either enabled limit. Limits are checked before the newline-appended Oniguruma input and its rune buffer are allocated.