lezer

Documentation

Index

• func Highlight(t *Tree, text []uint16, sets *TagSets, span func(from, to int, set *TagSet))
• func StyleTags(specs []StyleSpec) (map[string][]Rule, error)
• type ChangedRange
• type Config
• type ContextTracker
• type DialectSpec
• type Externals
• type Input
  • func (in Input) Read(from, to int) string
• type InputStream
  • func (in *InputStream) AcceptToken(term, endOffset int)
  • func (in *InputStream) AcceptTokenTo(term, endPos int)
  • func (in *InputStream) Advance(n int) int
  • func (in *InputStream) Peek(offset int) int
• type LocalGroupSpec
• type Mode
• type Nest
• type NestedParse
• type Node
  • func (n *Node) FirstChild() *Node
  • func (n *Node) From() int
  • func (n *Node) LastChild() *Node
  • func (n *Node) NextSibling() *Node
  • func (n *Node) Parent() *Node
  • func (n *Node) To() int
  • func (n *Node) Type() *NodeType
• type NodeType
  • func (t *NodeType) Anonymous() bool
• type Parser
  • func New(spec Spec, ext Externals) (*Parser, error)
  • func (p *Parser) Configure(c Config) (*Parser, error)
  • func (p *Parser) Parse(text string) *Tree
  • func (p *Parser) ParseUnits(input []uint16, ranges []Range, fragments []TreeFragment) *Tree
  • func (p *Parser) Types() []NodeType
  • func (p *Parser) WithNest(nest Nest) *Parser
  • func (p *Parser) WithStyleTags(byName map[string][]Rule) *Parser
• type Range
• type Rule
• type RuleSpec
• type Spec
• type Specializer
• type SpecializerSpec
• type Stack
  • func (s *Stack) CanShift(term int) bool
  • func (s *Stack) Context() any
  • func (s *Stack) DialectEnabled(id int) bool
  • func (s *Stack) Pos() int
• type StyleSpec
• type TagSet
• type TagSets
  • func NewTagSets() *TagSets
  • func (s *TagSets) Of(ts ...*tags.Tag) *TagSet
• type Tokenizer
• type TokenizerSpec
• type TopRule
• type Tree
  • func (t *Tree) Layout() string
  • func (t *Tree) Length() int
  • func (t *Tree) String() string
  • func (t *Tree) Type() *NodeType
• type TreeFragment
  • func AddTree(tree *Tree, fragments []TreeFragment, partial bool) []TreeFragment
  • func ApplyChanges(fragments []TreeFragment, changes []ChangedRange) []TreeFragment

Functions

func Highlight

func Highlight(t *Tree, text []uint16, sets *TagSets, span func(from, to int, set *TagSet))

Highlight calls span, in order, for every piece of t whose TagSet is not empty, merging neighbors with the same set, as highlightTree does with a highlighter that gives every tag its own class. text is what t was parsed from, in UTF-16 code units, which rules with Names read.

func StyleTags

func StyleTags(specs []StyleSpec) (map[string][]Rule, error)

StyleTags is what @lezer/highlight's styleTags makes of specs: each node name's rules, deepest context first. A path that does not parse is an error naming it.

Types

type ChangedRange

type ChangedRange struct {
	FromA, ToA, FromB, ToB int
}

ChangedRange is one edit between two versions of a document: [FromA, ToA) in the old one became [FromB, ToB) in the new one.

type Config

type Config struct {
	Top     string
	Dialect string
}

Config changes what a Configure'd parser parses: Top names one of the grammar's top rules ("" keeps the current one), Dialect is a space-separated list of dialects to enable.

type ContextTracker

type ContextTracker struct {
	Start     any
	Shift     func(context any, term int, s *Stack, in *InputStream) any
	Reduce    func(context any, term int, s *Stack, in *InputStream) any
	Reuse     func(context any, node *Tree, s *Stack, in *InputStream) any
	Hash      func(context any) int
	NonStrict bool
}

ContextTracker keeps a value along a parse that external tokenizers can read, such as the indentation in Python's grammar, the way @lezer/lr's ContextTracker does. Values are compared with ==, so they must be comparable; like @lezer/lr's they should be immutable, replaced rather than changed. Shift, Reduce and Reuse (for a node an incremental parse takes from an earlier tree) default to keeping the value. Unless NonStrict, Hash tells values apart for the token cache and the tree.

type DialectSpec

type DialectSpec struct {
	Name   string
	Offset int
}

DialectSpec is a dialect and the offset in the state data of the terms it alone enables.

type Externals

type Externals struct {
	Tokenizers   map[string]Tokenizer
	Specializers map[string]Specializer
	Context      *ContextTracker
}

Externals are the Go ports of what a grammar takes from its own JavaScript module: external tokenizers and specializers by export name, and its @context tracker.

type Input

type Input struct {
	// contains filtered or unexported fields
}

Input is the text a parse reads, for a Nest to look at.

func (Input) Read

func (in Input) Read(from, to int) string

Read is the text of [from, to), in UTF-16 code units, clipped to the input as JavaScript's String.slice clips it, so "" when to <= from.

type InputStream

type InputStream struct {

	// Next is the code unit at Pos, -1 at the end of the input.
	Next int
	// Pos is the stream's position in the input.
	Pos int
	// contains filtered or unexported fields
}

InputStream is how a tokenizer reads the input: as UTF-16 code units, the way JavaScript strings index, across the parse's ranges as if they were one text. The whole text is in memory, so @lezer/lr's chunking has no counterpart here.

func (*InputStream) AcceptToken

func (in *InputStream) AcceptToken(term, endOffset int)

AcceptToken accepts a token of type term ending endOffset units from Pos. An end before the token's start is a tokenizer bug, and panics.

func (*InputStream) AcceptTokenTo

func (in *InputStream) AcceptTokenTo(term, endPos int)

AcceptTokenTo accepts a token of type term ending at endPos.

func (*InputStream) Advance

func (in *InputStream) Advance(n int) int

Advance moves the stream n code units forward and returns the new Next.

func (*InputStream) Peek

func (in *InputStream) Peek(offset int) int

Peek is the code unit offset units from Pos, -1 outside the input.

type LocalGroupSpec

type LocalGroupSpec struct {
	Data      string
	PrecTable int
	ElseToken int // -1 for none
}

LocalGroupSpec is a LocalTokenGroup's constructor arguments.

type Mode

type Mode int

Mode is how a styleTags rule applies below its node, with the values @lezer/highlight gives them.

const (
	// ModeOpaque, a path ending in "!", styles the whole node and stops
	// matching inside it.
	ModeOpaque Mode = iota
	// ModeInherit, a path ending in "/...", adds the tags to every node
	// inside as well.
	ModeInherit
	// ModeNormal styles the node, its children matching their own rules.
	ModeNormal
)

type Nest

type Nest func(n *Node, in Input) *NestedParse

Nest is what @lezer/common's parseMixed takes: for a node of the outer parse, the parse of its content, or nil to leave it as it is.

type NestedParse

type NestedParse struct {
	Parser *Parser
	// Overlay, unless nil, are the parts of the input the parser reads, as
	// absolute positions inside the node, and the node keeps its own tree
	// around them. A nil Overlay parses the whole node, and its tree
	// replaces the node's.
	Overlay []Range
}

NestedParse says how another parser reads a node's content.

type Node

type Node struct {
	// contains filtered or unexported fields
}

Node is a node of a tree in place, as @lezer/common's SyntaxNode: a Tree, or a node inside a TreeBuffer. Its navigation skips anonymous nodes and enters mounted trees that replace their node.

func (*Node) FirstChild

func (n *Node) FirstChild() *Node

FirstChild is the node's first child, nil for none.

func (*Node) From

func (n *Node) From() int

From is where the node starts, in UTF-16 code units.

func (*Node) LastChild

func (n *Node) LastChild() *Node

LastChild is the node's last child, nil for none.

func (*Node) NextSibling

func (n *Node) NextSibling() *Node

NextSibling is the node after this one in its parent, nil for none.

func (*Node) Parent

func (n *Node) Parent() *Node

Parent is the nearest ancestor that is not anonymous, nil for the root.

func (*Node) To

func (n *Node) To() int

To is where the node ends, in UTF-16 code units.

func (*Node) Type

func (n *Node) Type() *NodeType

Type is the node's type.

type NodeType

type NodeType struct {
	Name  string
	ID    int
	Props map[string]string
	Top   bool
	Error bool
	// Skipped marks nodes the grammar's @skip produces, like comments.
	Skipped bool
	// Rules are the styleTags rules for the type, deepest context first.
	Rules []Rule
}

NodeType is a kind of node in a syntax tree.

func (*NodeType) Anonymous

func (t *NodeType) Anonymous() bool

Anonymous reports whether t has no name, as the repeat nodes lezer-generator adds for a `*` or `+`: a cursor walks through an anonymous Tree to its children.

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

Parser holds a grammar's parse tables and parses text with them, as @lezer/lr's LRParser does. It is immutable and safe to share between goroutines.

func New

func New(spec Spec, ext Externals) (*Parser, error)

New builds the parser for spec, taking the externals it names from ext. A missing external, a table of another format or a tag Parse rejects is an error.

func (*Parser) Configure

func (p *Parser) Configure(c Config) (*Parser, error)

Configure is a copy of p with c applied. An unknown top rule is an error.

func (*Parser) Parse

func (p *Parser) Parse(text string) *Tree

Parse parses text whole and returns its tree. Positions in the tree are UTF-16 code units, as in JavaScript.

func (*Parser) ParseUnits

func (p *Parser) ParseUnits(input []uint16, ranges []Range, fragments []TreeFragment) *Tree

ParseUnits parses the ranges of input, nil ranges meaning all of it, reusing what it can of fragments, the pieces of earlier trees ApplyChanges left: an incremental parse, as Parser.parse with fragments. A reused node is shared with the tree it came from, and a nested parse may store its tree on it, so no earlier tree may be read while this runs.

func (*Parser) Types

func (p *Parser) Types() []NodeType

Types are the parser's node types, indexed by id.

func (*Parser) WithNest

func (p *Parser) WithNest(nest Nest) *Parser

WithNest is a copy of p whose parses run nest over every node of the tree they made and mount the nested parses it asks for, as LRParser.configure with a wrap of parseMixed(nest) does. A nest p had is replaced.

func (*Parser) WithStyleTags

func (p *Parser) WithStyleTags(byName map[string][]Rule) *Parser

WithStyleTags is p with byName's rules added to its node types' own, as configuring a parser with a styleTags prop does: a node type takes the rules of its name, or else of the first of its groups that has some. Trees p parsed are not reused by the new parser's incremental parses, since they hold p's node types.

type Range

type Range struct{ From, To int }

Range is a span of the input, in UTF-16 code units.

type Rule

type Rule struct {
	Tags    []*tags.Tag
	Mode    Mode
	Context []string
	Names   map[string]bool
}

Rule is one styleTags rule of a node type: its tags, the names of the direct parents the node must have for it to apply ("" for any), and the texts it is limited to (nil for any).

type RuleSpec

type RuleSpec struct {
	Tags    []string
	Mode    Mode
	Context []string
}

RuleSpec is one styleTags rule as lezer-generator's caller wrote it: tags as Tag.toString prints them.

type Spec

type Spec struct {
	Version         int
	States          string
	StateData       string
	Goto            string
	NodeNames       string
	MaxTerm         int
	RepeatNodeCount int
	NodeProps       map[string]map[int]string
	SkippedNodes    []int
	TokenData       string
	Tokenizers      []TokenizerSpec
	TopRules        []TopRule
	Dialects        []DialectSpec
	DynamicPrec     map[int]int
	Specialized     []SpecializerSpec
	TokenPrec       int
	// Context names the grammar's @context tracker, "" for none.
	Context string
	// Styles are each node type's styleTags rules, already combined and
	// sorted the way @lezer/highlight's ruleNodeProp leaves them.
	Styles map[int][]RuleSpec
}

Spec is a grammar's parse tables, as lezer-generator writes them for LRParser.deserialize, plus the styleTags rules its language package attaches. internal/lezer/gen writes one per grammar.

type Specializer

type Specializer func(text string, s *Stack) int

Specializer maps a token's text to the term it specializes to, or -1 to leave the token as it is. Whether that term replaces the token or is tried alongside it is the grammar's @external specialize or extend.

type SpecializerSpec

type SpecializerSpec struct {
	Term     int
	Table    map[string]int
	External string
	Extend   bool
}

SpecializerSpec is a @specialize or @extend on Term: a table from token text to the encoded result (term<<1, |1 for an extension), or an external specializer by its export name.

type Stack

type Stack struct {
	// contains filtered or unexported fields
}

Stack is one parse stack: a parse runs several when the grammar is ambiguous or while it recovers from an error. Tokenizers and context trackers read the parse state through it.

func (*Stack) CanShift

func (s *Stack) CanShift(term int) bool

CanShift reports whether term could be shifted on the stack, after the reductions it would take: for an external tokenizer that should emit a token only where it applies.

func (*Stack) Context

func (s *Stack) Context() any

Context is the stack's context tracker value, nil without a tracker.

func (*Stack) DialectEnabled

func (s *Stack) DialectEnabled(id int) bool

DialectEnabled reports whether the dialect with the given index (its place in the grammar's dialect list) is enabled.

func (*Stack) Pos

func (s *Stack) Pos() int

Pos is the input position the stack has parsed up to.

type StyleSpec

type StyleSpec struct {
	Selector string
	Tags     []*tags.Tag
	Names    []string
}

StyleSpec is one entry of a styleTags object: space-separated paths ("Block/...", "Annotation!", "*/ArgumentList/Identifier") and the tags the nodes they name take. Names, when set, limit the entry to nodes whose text is one of them, which styleTags has no way to say.

type TagSet

type TagSet struct {
	Tags []*tags.Tag
}

TagSet is the tags that apply to a piece of text: a node's own and the ones inherited from its ancestors, in that order, as the class string highlightTree builds lists them. TagSets from one TagSets are interned, so two are equal exactly when they hold the same list.

type TagSets

type TagSets struct {
	// contains filtered or unexported fields
}

TagSets interns TagSets. It is not safe for concurrent use.

func NewTagSets

func NewTagSets() *TagSets

NewTagSets returns an empty TagSets.

func (*TagSets) Of

func (s *TagSets) Of(ts ...*tags.Tag) *TagSet

Of is the interned set holding ts, nil for none.

type Tokenizer

type Tokenizer struct {
	Token                        func(in *InputStream, s *Stack)
	Contextual, Fallback, Extend bool
}

Tokenizer reads a token at the stream's position and accepts it with AcceptToken, as @lezer/lr's ExternalTokenizer does. Contextual tokens are re-read in every parse state instead of cached by position; a Fallback tokenizer runs even after an earlier one produced a token; an Extend tokenizer's token is tried alongside the next tokenizer's.

type TokenizerSpec

type TokenizerSpec struct {
	Group    int
	Local    *LocalGroupSpec
	External string
}

TokenizerSpec is one entry of a grammar's tokenizers: a token group of its TokenData, a local token group with its own data, or an external tokenizer by its export name.

type TopRule

type TopRule struct {
	Name        string
	State, Term int
}

TopRule is one of a grammar's top rules, in the order the grammar lists them: the first is the default.

type Tree

type Tree struct {
	// contains filtered or unexported fields
}

Tree is a syntax tree node with its children, as @lezer/common builds it: small subtrees are packed into TreeBuffers, and long repetitions are balanced under anonymous nodes. The layout is kept exactly, because a tree cursor's view of a node's context depends on it (see cursor.go). Positions are UTF-16 code units, relative to the tree's start.

func (*Tree) Layout

func (t *Tree) Layout() string

Layout describes how the tree is stored, which a node an incremental parse reuses keeps from the tree it came from, as gen/incremental.mjs writes it for @lezer/common's trees: every Tree as its type id and length, its contextHash (#) and lookAhead (~), a mounted tree in braces after its overlay ranges, and its children in brackets, each after its position, a TreeBuffer as B<length>/<buffer size>.

func (*Tree) Length

func (t *Tree) Length() int

Length is the length of the text the tree covers.

func (*Tree) String

func (t *Tree) String() string

String prints the tree as @lezer/common's Tree.toString does, as in `Program(ClassDeclaration(class,Definition,ClassBody("{","}")))`.

func (*Tree) Type

func (t *Tree) Type() *NodeType

Type is the tree's node type.

type TreeFragment

type TreeFragment struct {
	From, To           int
	Tree               *Tree
	Offset             int
	OpenStart, OpenEnd bool
}

TreeFragment is a piece of an earlier tree an incremental parse may reuse: the unchanged range [From, To) of the new document, and the Offset to add to go from a new position to the tree's. OpenStart and OpenEnd say whether the fragment's ends are the ends of a change rather than of the parse, where nodes may not be safe to reuse.

func AddTree

func AddTree(tree *Tree, fragments []TreeFragment, partial bool) []TreeFragment

AddTree is the fragments of a fresh tree, with the fragments that reach past its end kept, as TreeFragment.addTree does. A partial tree's fragment is open at its end.

func ApplyChanges

func ApplyChanges(fragments []TreeFragment, changes []ChangedRange) []TreeFragment

ApplyChanges drops or cuts the fragments the changes touch and moves the ones after them, as TreeFragment.applyChanges does with its default minGap. changes are in order and do not overlap.