textmate

README

textmate-go

textmate-go is a pure-Go port of Microsoft's vscode-textmate. It provides line-oriented TextMate grammar tokenization with an immutable state stack that can be carried between lines.

The port is under active development, so its API may still evolve before a stable release. It targets Go 1.25 or newer and builds with CGO_ENABLED=0.

Reference revisions

Development and conformance testing use sibling checkouts at these revisions:

• vscode-textmate: fbe49961ab8077e587fdf5282019655ae69e5f9e
• textmate-grammars-themes: 37edd1b26f18838050661d912334aba0ca7f4931
• gopher-textmate: ce2b42e5386c93ae781add9df2e6328338b06f9e
• regexp2: maintained fork github.com/eugenioenko/regexp2/v2, release v2.8.1 (upstream v2.8.0 base 9d0d2ffe88a8b90012f7979ec85424e46d5ef48f)

The implementation is ported from vscode-textmate; its Microsoft MIT license is reproduced in THIRD_PARTY_NOTICES.

Regex engine

TextMate grammars require lookbehind, backreferences, and position-sensitive anchors that Go's standard RE2-based regexp package cannot provide. This port uses github.com/dlclark/regexp2/v2 v2.8.0 with RE2 compatibility mode disabled. The v2 line was selected for ordered mixed captures, current Unicode tables, bounded backtracking, and its Go 1.25 API. Oniguruma-only syntax is translated before compilation; unsupported constructs are recorded as diagnostics and the affected construct or pattern is safely degraded.

The current performance branch uses the maintained github.com/eugenioenko/regexp2/v2 fork with a text-free capture-index API. This avoids constructing public match/group data that the tokenizer immediately discarded. The fork has its own module path, so downstream users receive it as a normal transitive dependency without a replace directive.

Development

make test
make bench
make bench-compare   # textmate-go vs vscode-textmate vs Chroma
make lint
CGO_ENABLED=0 go build ./...

make bench-compare needs a compiled vscode-textmate checkout and the conformance harness's pnpm install; see benchmarks/README.md for setup, options, and the corpus.

Embedded grammars

The optional grammars subpackage embeds 124 license-reviewed grammar roots plus one MIT support grammar used for Markdown's inline HTML. Assets are individually compressed and loaded once on demand:

import (
	"log"
	"time"

	textmate "github.com/eugenioenko/textmate-go"
	"github.com/eugenioenko/textmate-go/grammars"
)

registry := textmate.NewRegistry(textmate.RegistryOptions{
	LoadGrammar: grammars.Load,
})
grammar, err := registry.LoadGrammar(grammars.ScopeForFilename("main.go"))
if err != nil {
	log.Fatal(err)
}
if grammar == nil {
	log.Fatal("no embedded grammar for main.go")
}
defer registry.Dispose()

var state *textmate.StateStack
for _, line := range []string{"package main", `const message = "hello"`} {
	result := grammar.TokenizeLine(line, state)
	for _, token := range result.Tokens {
		// token.Start and token.End are rune offsets; token.Scopes is ordered
		// from the root grammar to the most specific matched scope.
		// ScopeStack and its ID are comparable cache keys for resolved styles.
		styleCacheKey := token.ScopeStack.ID()
		_ = styleCacheKey
		_ = token
	}
	state = result.RuleStack
}

// Unsupported Oniguruma constructs, compile failures, and match timeouts are
// non-fatal and can be inspected after lazy compilation/tokenization.
for _, diagnostic := range grammar.Diagnostics() {
	_ = diagnostic
}

A state stack is immutable and reusable, but belongs to the grammar that created it. Pass nil or textmate.InitialState for the first line. Equal states returned by the same grammar have the same pointer while both are live, so consumers may use *StateStack as a short-lived cache key and stop incremental highlighting with a pointer comparison. The grammar holds only a weak reference to canonical states: if every returned copy is discarded, a later equal state may receive a new pointer. Pointer identity is not shared between grammars, and states must not be carried from one grammar into another. Use StateStack.Equal when a previously returned equal pointer is no longer being retained.

Editable documents

Document owns the state table and token cache needed to highlight an editable buffer without replaying it from the beginning for every visible line:

document := textmate.NewDocument(grammar, textmate.DocumentOptions{
	TokenizeOptions: textmate.TokenizeOptions{
		MaxLineBytes: 20_000,
		TimeLimit:    5 * time.Millisecond,
	},
	CacheCapacity: 20_000,
})
document.SetLines([]string{"package main", `const message = "hello"`})

line, ok := document.Line(1) // computes any missing states above line 1
if ok {
	for _, token := range line.Tokens {
		_ = token
	}
}

// Replace the half-open range [1, 2). Common-prefix states are retained.
// The unchanged tail is reattached as soon as its canonical state converges.
if err := document.ReplaceLines(1, 2, []string{`const message = "goodbye"`}); err != nil {
	log.Fatal(err)
}

// Explicitly discard state from a line onward when retrying work that stopped
// under a transient time budget.
if err := document.InvalidateFrom(1); err != nil {
	log.Fatal(err)
}

SetLines and ReplaceLines shallow-copy their input slices. Line, StateAt, Len, and updates are safe to call concurrently. StateAt also accepts document.Len() to obtain the state after the final line.

Line results are immutable, library-owned views and remain valid after later calls or cache eviction. This makes warmed cache hits allocation-free. Copy LineResult.Tokens before changing token fields; Token.Scopes and Token.ScopeStack remain read-only. Cache keys combine line text with the canonical start-state pointer, so identical lines in identical states reuse a result even after edits. CacheCapacity bounds retained results, with zero selecting the 20,000-entry default and a negative value disabling the cache.

Line-length stops are deterministic for a document's fixed options and are cached normally. Time-limit stops are transient: Document materializes their outgoing state so a tiny budget cannot trap a random-access request in a retry loop, but it never caches their line result. Calling Line again retries that line. If a successful retry changes its outgoing state, the document truncates stale downstream states and reconciles the unchanged tail again. InvalidateFrom(i) explicitly discards materialized state from line i onward and clears the result cache, which is useful before retrying a viewport whose earlier state was produced by a time-limited line. Passing Len() is valid and clears the cache without discarding an existing line state.

Interned token scopes

Every token also carries an immutable *ScopeStack. Equal scope sequences produced by one grammar reuse the same pointer and ScopeStackID, even when the tokenizer reconstructed its internal state. Consumers can therefore cache resolved styles by token.ScopeStack or token.ScopeStack.ID() instead of hashing strings for every token. The handle remains valid independently of the grammar and its accessors are safe for concurrent reads. Names returns a defensive copy; Len, At, and Range inspect names without allocating.

The legacy Token.Scopes field remains available and preserves existing JSON and conformance output. Equal stacks share its backing array, so it must be treated as read-only. Copy the slice before modifying it.

Token categories

For consumers that need semantic classes rather than full TextMate theme resolution, each token and scope stack exposes a cached coarse category:

switch token.Category() {
case textmate.TokenCategoryComment:
	// Apply the application's comment style.
case textmate.TokenCategoryInserted:
	// Apply the application's added-line style.
}

textmate.ClassifyScopes(scopes) provides the same classification for a raw scope-name slice. It recognizes common comments, strings, regular expressions, numbers, keywords, operators, functions, tags, attributes, types, builtins, variables, markup, diff, punctuation, and invalid scopes. The innermost recognized scope wins, while reset scopes such as meta.embedded prevent an outer string from coloring embedded source. This is deliberately a stable, theme-neutral heuristic; it is not a replacement for TextMate selector and theme resolution.

Tokenization limits

Editors can bound work on untrusted, generated, or minified input without changing the behavior of TokenizeLine:

result := grammar.TokenizeLineWithOptions(line, state, textmate.TokenizeOptions{
	MaxLineBytes: 20_000,
	MaxLineRunes: 10_000,
	TimeLimit:    5 * time.Millisecond,
})
if result.Stopped {
	log.Printf("tokenization stopped at rune %d: %s", result.StoppedAt, result.StoppedReason)
}

Zero or negative limits are unlimited. A line that exceeds either size cap is not parsed: it receives one fallback token with the incoming scopes, its reset incoming state is returned unchanged, and StoppedAt is zero. Size caps are checked before allocating the newline-appended regexp input and rune buffer.

TimeLimit is deliberately a soft overall budget. It includes lock wait, lazy root compilation, and begin/while setup, and is checked between regexp searches. The library cannot interrupt an in-flight regexp search, so a call can return after the requested duration. A partial result reports the next unparsed rune in StoppedAt, carries the state reached at the last completed match boundary, and gives the unparsed tail a fallback token so the returned tokens still cover the complete line.

See grammars/README.md for the curated selection, regeneration instructions, source/license manifest, and all-versus-curated size measurements.

Performance

Warm line-by-line tokenization from make bench-compare, in microseconds per line (fastest of five runs, Linux/amd64, Go 1.25.1, AMD Ryzen 7 6800H). The ratio is textmate-go's time over vscode-textmate's, so below 1 is faster. Chroma lexes each line from its root state, so its column is a throughput reference rather than an equivalent result. See benchmarks/README.md for the method and corpus.

Case	textmate-go	allocs/line	vscode-textmate	Chroma	vs vscode-textmate
TSX	33.7	42.6	19.6	17.0	1.71x
HTML	24.5	62.4	25.3	5.2	0.97x
Go	9.1	31.3	13.6	19.6	0.67x
Markdown	15.3	29.3	13.6	18.6	1.13x
TypeScript	85.7	74.0	55.6	25.5	1.54x
JavaScript	77.6	75.3	61.9	26.0	1.25x
CSS	32.5	53.7	71.1	11.1	0.46x
JSON	9.1	52.8	7.1	9.0	1.28x
Python	46.2	71.1	46.6	53.6	0.99x
Rust	27.3	48.6	30.5	29.2	0.89x
Java	60.8	57.3	38.4	34.6	1.58x
C++	272.0	106.5	145.9	44.3	1.86x
Ruby	40.9	51.1	56.0	62.6	0.73x
Shell	27.8	86.7	30.5	22.2	0.91x

C++ is the slowest case: its grammar's very large patterns backtrack heavily in regexp2's interpreter. The earlier Phase 4 baseline is kept in docs/performance-baseline.md.

See plan.md for the implementation phases and conformance goals.

License

MIT. See LICENSE.

Documentation

Overview

Package textmate provides pure-Go, line-oriented tokenization using TextMate grammars. Token offsets are rune indices. Callers pass each LineResult's immutable RuleStack into the next line and may compare saved stacks with StateStack.Equal when incrementally re-tokenizing edited documents.

Index

• Constants
• Variables
• type Document
  • func NewDocument(grammar *Grammar, options DocumentOptions) *Document
  • func (d *Document) InvalidateFrom(index int) error
  • func (d *Document) Len() int
  • func (d *Document) Line(index int) (LineResult, bool)
  • func (d *Document) ReplaceLines(start, end int, replacement []string) error
  • func (d *Document) SetLines(lines []string)
  • func (d *Document) StateAt(index int) (*StateStack, bool)
• type DocumentOptions
• type Grammar
  • func (g *Grammar) Diagnostics() []oniguruma.Diagnostic
  • func (g *Grammar) TokenizeLine(line string, prev *StateStack) LineResult
  • func (g *Grammar) TokenizeLineWithOptions(line string, prev *StateStack, options TokenizeOptions) LineResult
• type IncludeString
• type LineResult
• type Location
• type RawCaptures
  • func (captures *RawCaptures) UnmarshalJSON(data []byte) error
• type RawGrammar
  • func ParseRawGrammar(data []byte) (*RawGrammar, error)
• type RawInjections
• type RawRepository
  • func (repository *RawRepository) UnmarshalJSON(data []byte) error
• type RawRule
  • func (r *RawRule) UnmarshalJSON(data []byte) error
• type RegExpString
• type Registry
  • func NewRegistry(options RegistryOptions) *Registry
  • func (r *Registry) Dispose()
  • func (r *Registry) LoadGrammar(scopeName string) (*Grammar, error)
• type RegistryOptions
• type RuleFactory
• type ScopeStack
  • func (s *ScopeStack) At(index int) (string, bool)
  • func (s *ScopeStack) Category() TokenCategory
  • func (s *ScopeStack) ID() ScopeStackID
  • func (s *ScopeStack) Len() int
  • func (s *ScopeStack) Names() []string
  • func (s *ScopeStack) Range(yield func(string) bool)
• type ScopeStackID
  • func (id ScopeStackID) IsZero() bool
• type StateStack
  • func (s *StateStack) Equal(other *StateStack) bool
  • func (s *StateStack) String() string
• type StopReason
  • func (r StopReason) String() string
• type Token
  • func (token Token) Category() TokenCategory
• type TokenCategory
  • func ClassifyScopes(scopes []string) TokenCategory
  • func (category TokenCategory) String() string
• type TokenizeOptions

Constants

const DefaultDocumentCacheCapacity = 20_000

DefaultDocumentCacheCapacity is the number of distinct line/state results retained by a Document when DocumentOptions.CacheCapacity is zero.

Variables

var ErrInvalidLineRange = errors.New("textmate: invalid document line range")

ErrInvalidLineRange is returned when ReplaceLines receives a range outside the current document or one whose start is after its end.

var InitialState = newStateStack(nil, 0, 0, 0, false, nil, nil, nil)

InitialState is the sentinel passed when tokenizing the first line.

Types

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document incrementally tokenizes an editable sequence of lines. It owns a shallow copy of the supplied line slice, carries start states between lines, and reuses an unchanged suffix after an edit as soon as tokenizer state converges with the previous suffix.

All methods are safe to call concurrently. Results returned by Line are immutable, library-owned views and must not be modified; this permits cache hits without allocating. A result remains valid after later calls and cache eviction. Copy the Tokens slice before changing Token fields. ScopeStack and Token.Scopes remain library-owned and read-only, as they are for Grammar.TokenizeLine.

func NewDocument

func NewDocument(grammar *Grammar, options DocumentOptions) *Document

NewDocument creates an empty document backed by grammar. A nil grammar is valid: lines remain addressable, but Line returns empty tokenization results.

func (*Document) InvalidateFrom

func (d *Document) InvalidateFrom(index int) error

InvalidateFrom discards materialized states for line index and all following lines, and clears the result cache. Index may equal Len. The next Line or StateAt call recomputes the discarded range.

Time-limit stops are never cached, so calling Line again retries that line directly. InvalidateFrom is useful when a stopped line has already supplied state to later lines and the caller wants the complete tail recomputed.

func (*Document) Len

func (d *Document) Len() int

Len returns the current number of lines.

func (*Document) Line

func (d *Document) Line(index int) (LineResult, bool)

Line returns the tokenization result for line index. The boolean is false when index is outside the current document. Requesting a line computes any missing start states above it and records the line's resulting state.

The returned result and its Tokens slice are immutable, library-owned views. They remain valid after later calls and cache eviction. See Document's ownership documentation.

func (*Document) ReplaceLines

func (d *Document) ReplaceLines(start, end int, replacement []string) error

ReplaceLines replaces the half-open line range [start, end) with a shallow copy of replacement. It returns ErrInvalidLineRange without changing the document when the range is invalid.

func (*Document) SetLines

func (d *Document) SetLines(lines []string)

SetLines replaces the complete document with a shallow copy of lines. Tokenizer states above the common prefix are retained. States in the common suffix become reusable once recomputation reaches an equal canonical state.

func (*Document) StateAt

func (d *Document) StateAt(index int) (*StateStack, bool)

StateAt returns the state in which line index starts. Index may equal Len, in which case StateAt returns the state immediately after the final line. The boolean is false for all other out-of-range indices.

type DocumentOptions

type DocumentOptions struct {
	// TokenizeOptions is applied to every line tokenized by the document.
	TokenizeOptions TokenizeOptions

	// CacheCapacity bounds cached results keyed by line text and canonical
	// start state. Zero selects DefaultDocumentCacheCapacity. A negative value
	// disables result caching.
	CacheCapacity int
}

DocumentOptions configures a Document. Options are fixed at construction.

type Grammar

type Grammar struct {
	// contains filtered or unexported fields
}

Grammar is a compiled TextMate grammar. Compilation remains lazy: rules and their regular expressions are created only when the grammar is first used. A Grammar serializes tokenization because compiled rules contain mutable scanner caches.

func (*Grammar) Diagnostics

func (g *Grammar) Diagnostics() []oniguruma.Diagnostic

Diagnostics returns regex translation, compilation, and match diagnostics from rules compiled or exercised so far. Grammars compile lazily, so callers should take another snapshot after tokenizing new syntax paths. Equivalent diagnostics produced by anchor variants or shared rule lists are returned once.

func (*Grammar) TokenizeLine

func (g *Grammar) TokenizeLine(line string, prev *StateStack) LineResult

TokenizeLine tokenizes line and returns immutable state to pass to the next call. A nil state and InitialState both begin a new document.

func (*Grammar) TokenizeLineWithOptions

func (g *Grammar) TokenizeLineWithOptions(
	line string,
	prev *StateStack,
	options TokenizeOptions,
) LineResult

TokenizeLineWithOptions tokenizes line with optional soft time and line-size limits. StoppedAt is the first unparsed rune offset when Stopped is true. See TokenizeOptions for the precise limit semantics.

type IncludeString

type IncludeString = string

IncludeString is a TextMate include reference such as "#rule", "$self", or "source.js#rule".

type LineResult

type LineResult struct {
	Tokens    []Token
	RuleStack *StateStack
	// Stopped remains the compatibility signal for an incomplete parse.
	Stopped bool
	// StoppedReason is StopReasonNone when Stopped is false.
	StoppedReason StopReason
	// StoppedAt is the first unparsed rune offset when Stopped is true.
	StoppedAt int
}

LineResult is the result of tokenizing one line.

type Location

type Location struct {
	Filename string `json:"filename"`
	Line     int    `json:"line"`
	Char     int    `json:"char"`
}

Location identifies a value in a source grammar. Plain JSON grammars do not normally contain it, but vscode-textmate can attach it while parsing.

type RawCaptures

type RawCaptures map[string]*RawRule

RawCaptures maps numeric capture group names to their rules.

func (*RawCaptures) UnmarshalJSON

func (captures *RawCaptures) UnmarshalJSON(data []byte) error

UnmarshalJSON accepts both the specified object form and the array form emitted by a few converted TextMate grammars. Invalid capture entries are ignored, matching JavaScript's permissive property access on those values.

type RawGrammar

type RawGrammar struct {
	Repository        RawRepository `json:"repository"`
	ScopeName         string        `json:"scopeName"`
	Patterns          []*RawRule    `json:"patterns"`
	Injections        RawInjections `json:"injections,omitempty"`
	InjectionSelector string        `json:"injectionSelector,omitempty"`
	FileTypes         []string      `json:"fileTypes,omitempty"`
	Name              string        `json:"name,omitempty"`
	FirstLineMatch    RegExpString  `json:"firstLineMatch,omitempty"`
	Location          *Location     `json:"$vscodeTextmateLocation,omitempty"`
}

RawGrammar is the JSON representation of a TextMate grammar.

func ParseRawGrammar

func ParseRawGrammar(data []byte) (*RawGrammar, error)

ParseRawGrammar decodes a JSON TextMate grammar.

type RawInjections

type RawInjections map[string]*RawRule

RawInjections maps scope selector expressions to injected rules.

type RawRepository

type RawRepository map[string]*RawRule

RawRepository maps grammar-defined repository names to rules.

func (*RawRepository) UnmarshalJSON

func (repository *RawRepository) UnmarshalJSON(data []byte) error

UnmarshalJSON ignores malformed non-object entries found in a small number of converted grammars. JavaScript loads these values but they do not compile into meaningful rules.

type RawRule

type RawRule struct {
	Include *IncludeString `json:"include,omitempty"`

	Name        *string `json:"name,omitempty"`
	ContentName *string `json:"contentName,omitempty"`

	Match         *RegExpString `json:"match,omitempty"`
	Captures      RawCaptures   `json:"captures,omitempty"`
	Begin         *RegExpString `json:"begin,omitempty"`
	BeginCaptures RawCaptures   `json:"beginCaptures,omitempty"`
	End           *RegExpString `json:"end,omitempty"`
	EndCaptures   RawCaptures   `json:"endCaptures,omitempty"`
	While         *RegExpString `json:"while,omitempty"`
	WhileCaptures RawCaptures   `json:"whileCaptures,omitempty"`
	Patterns      []*RawRule    `json:"patterns,omitempty"`

	Repository          RawRepository `json:"repository,omitempty"`
	ApplyEndPatternLast bool          `json:"applyEndPatternLast,omitempty"`
	Location            *Location     `json:"$vscodeTextmateLocation,omitempty"`
}

RawRule is one rule in a TextMate grammar.

func (*RawRule) UnmarshalJSON

func (r *RawRule) UnmarshalJSON(data []byte) error

type RegExpString

type RegExpString = string

RegExpString is a regular expression in the grammar's Oniguruma dialect.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry owns raw and compiled grammars and resolves their transitive dependencies. Callback invocations never occur while the registry lock is held.

func NewRegistry

func NewRegistry(options RegistryOptions) *Registry

NewRegistry constructs an empty lazy-loading grammar registry.

func (*Registry) Dispose

func (r *Registry) Dispose()

Dispose releases the registry's references to loaded grammars. It is idempotent; subsequent loads fail.

func (*Registry) LoadGrammar

func (r *Registry) LoadGrammar(scopeName string) (*Grammar, error)

LoadGrammar loads scopeName and all reachable includes and injections, then returns the cached compiled grammar for scopeName.

type RegistryOptions

type RegistryOptions struct {
	LoadGrammar   func(scopeName string) (*RawGrammar, error)
	GetInjections func(scopeName string) []string
}

RegistryOptions provides the callbacks used to lazily resolve grammars. A loader may return (nil, nil) when a scope is not available. Missing root grammars are errors; missing optional dependencies are ignored.

type RuleFactory

type RuleFactory struct {
	// contains filtered or unexported fields
}

RuleFactory turns raw grammar nodes into registered rules. A factory belongs to one compiled Grammar; keeping raw node identity here avoids mutating the public JSON model with compilation state.

type ScopeStack

type ScopeStack struct {
	// contains filtered or unexported fields
}

ScopeStack is an immutable, interned sequence of TextMate scope names. Equal scope sequences produced by one Grammar have the same *ScopeStack and ScopeStackID, including when the tokenizer reconstructed its internal stack. A ScopeStack remains valid after its Grammar or Registry is no longer used.

ScopeStack values are owned by the library and must not be modified. Its accessors are safe to call concurrently.

func (*ScopeStack) At

func (s *ScopeStack) At(index int) (string, bool)

At returns the scope name at index and whether index is in range.

func (*ScopeStack) Category

func (s *ScopeStack) Category() TokenCategory

Category returns the cached coarse category of s. A nil ScopeStack is plain. The first call computes the result; subsequent calls are lock-free and allocate no memory.

func (*ScopeStack) ID

func (s *ScopeStack) ID() ScopeStackID

ID returns the stable runtime identity of s. A nil ScopeStack has the zero ID.

func (*ScopeStack) Len

func (s *ScopeStack) Len() int

Len returns the number of scope names in s.

func (*ScopeStack) Names

func (s *ScopeStack) Names() []string

Names returns a fresh copy of the scope names, ordered outermost first.

func (*ScopeStack) Range

func (s *ScopeStack) Range(yield func(string) bool)

Range calls yield for each scope name, ordered outermost first. Iteration stops when yield returns false.

type ScopeStackID

type ScopeStackID struct {
	// contains filtered or unexported fields
}

ScopeStackID identifies one interned scope-name stack. IDs are comparable and unique within the current process, so they can be used directly as map keys. The zero value does not identify a scope stack.

An ID remains valid for as long as the ScopeStack that returned it is reachable. IDs are runtime identities and must not be persisted.

func (ScopeStackID) IsZero

func (id ScopeStackID) IsZero() bool

IsZero reports whether id is the zero value.

type StateStack

type StateStack struct {
	// contains filtered or unexported fields
}

StateStack is an immutable tokenizer state. A returned state can safely be retained and compared after subsequent lines have been tokenized.

While two equal states returned by the same Grammar are both live, they are represented by the same *StateStack. This makes pointers suitable as short-lived cache keys. The identity is scoped to one Grammar and lasts only as long as the state remains reachable; use Equal after discarding all previously returned equal states. States are not transferable between grammars, and no pointer identity is promised across Grammar instances.

func (*StateStack) Equal

func (s *StateStack) Equal(other *StateStack) bool

Equal reports whether two stacks represent the same reusable tokenizer state. Line-local enter and anchor positions are intentionally ignored, as they are by vscode-textmate.

func (*StateStack) String

func (s *StateStack) String() string

type StopReason

type StopReason uint8

StopReason explains why a LineResult stopped before fully parsing its line.

const (
	// StopReasonNone means the line was fully tokenized.
	StopReasonNone StopReason = iota
	// StopReasonTimeLimit means the soft time budget was exhausted.
	StopReasonTimeLimit
	// StopReasonLineLimit means a byte or rune line-length cap was exceeded.
	StopReasonLineLimit
)

func (StopReason) String

func (r StopReason) String() string

type Token

type Token struct {
	Start      int
	End        int
	Scopes     []string
	ScopeStack *ScopeStack `json:"-"`
}

Token is one contiguous span of a tokenized line. Start and End are rune offsets into the original line, and Scopes is ordered outermost first.

ScopeStack is the immutable, comparable identity for Scopes. Equal scope sequences from the same Grammar reuse both ScopeStack and the backing array of Scopes. Scopes is retained for source compatibility and must be treated as read-only; callers that need to modify it should copy it first.

func (Token) Category

func (token Token) Category() TokenCategory

Category returns the token's coarse semantic category. It uses the cached ScopeStack classification when available and falls back to Scopes for tokens constructed by callers or older integrations.

type TokenCategory

type TokenCategory uint8

TokenCategory is a coarse semantic classification derived from TextMate scope names. It is intentionally independent of colors and font styles. Applications can map categories to their own presentation or use a full TextMate theme resolver when selector-accurate styling is required.

const (
	TokenCategoryPlain TokenCategory = iota
	TokenCategoryComment
	TokenCategoryString
	TokenCategoryRegexp
	TokenCategoryNumber
	TokenCategoryKeyword
	TokenCategoryOperator
	TokenCategoryFunction
	TokenCategoryTag
	TokenCategoryAttribute
	TokenCategoryType
	TokenCategoryBuiltin
	TokenCategoryVariable
	TokenCategoryHeading
	TokenCategoryBold
	TokenCategoryItalic
	TokenCategoryQuote
	TokenCategoryInserted
	TokenCategoryDeleted
	TokenCategoryPunctuation
	TokenCategoryInvalid
)

func ClassifyScopes

func ClassifyScopes(scopes []string) TokenCategory

ClassifyScopes returns a coarse category for a TextMate scope stack ordered outermost first. The innermost recognized scope wins. Within one scope, the longest prefix at a dot boundary wins. Plain is returned when no scope is recognized and for deliberate reset scopes such as meta.embedded.

Classification is a heuristic semantic layer, not TextMate theme resolution. The function does not retain or modify scopes and allocates no memory.

func (TokenCategory) String

func (category TokenCategory) String() string

String returns the stable, lower-case name of category.

type TokenizeOptions

type TokenizeOptions struct {
	TimeLimit    time.Duration
	MaxLineBytes int
	MaxLineRunes int
}

TokenizeOptions bounds work performed by TokenizeLineWithOptions. Zero or negative values disable the corresponding limit.

TimeLimit is a soft budget for the complete call, including waiting for the grammar lock, lazy root compilation, and begin/while condition setup. The tokenizer checks the budget between regexp searches; an in-flight regexp search cannot be interrupted and can therefore overrun the budget.

MaxLineBytes and MaxLineRunes are independent. A line is skipped when it exceeds either enabled limit. Limits are checked before the newline-appended Oniguruma input and its rune buffer are allocated.