Documentation
¶
Index ¶
- Constants
- func DecodeUTF8(s []byte) (int32, uint32)
- func DecodeUTF16BE(s []byte) (int32, uint32)
- func DecodeUTF16LE(s []byte) (int32, uint32)
- func EncodeTables(t *Tables) ([]byte, error)
- func SetContains(ranges []CharacterRange, lookahead int32) bool
- type CharacterRange
- type ExternalScanner
- type FieldID
- type FieldMapEntry
- type Input
- type InputEdit
- type InputEncoding
- type Language
- func (l *Language) AliasAt(productionID uint32, childIndex uint32) Symbol
- func (l *Language) AliasesForSymbol(originalSymbol Symbol) []Symbol
- func (l *Language) FieldIDForName(name string) FieldID
- func (l *Language) FieldNameForID(id FieldID) string
- func (l *Language) LanguageName() string
- func (l *Language) StateIsPrimary(state StateID) bool
- func (l *Language) Subtypes(supertype Symbol) []Symbol
- func (l *Language) Supertypes() []Symbol
- func (l *Language) SymbolCountTotal() uint32
- func (l *Language) SymbolForName(name string, isNamed bool) Symbol
- func (l *Language) SymbolName(symbol Symbol) string
- type LanguageMetadata
- type LexInstr
- type LexOp
- type LexProgram
- type Lexer
- type LexerMode
- type MapSlice
- type Node
- func (n Node) Child(index uint32) Node
- func (n Node) ChildByFieldID(fieldID FieldID) Node
- func (n Node) ChildByFieldName(name string) Node
- func (n Node) ChildCount() uint32
- func (n Node) ChildWithDescendant(descendant Node) Node
- func (n Node) Children() []Node
- func (n Node) DescendantCount() uint32
- func (n Node) DescendantForByteRange(start, end uint32) Node
- func (n Node) DescendantForPointRange(start, end Point) Node
- func (n *Node) Edit(edit InputEdit)
- func (n Node) EndByte() uint32
- func (n Node) EndPoint() Point
- func (n Node) Equal(other Node) bool
- func (n Node) FieldNameForChild(index uint32) string
- func (n Node) FieldNameForNamedChild(index uint32) string
- func (n Node) GrammarSymbol() Symbol
- func (n Node) GrammarType() string
- func (n Node) HasChanges() bool
- func (n Node) HasError() bool
- func (n Node) IsError() bool
- func (n Node) IsExtra() bool
- func (n Node) IsMissing() bool
- func (n Node) IsNamed() bool
- func (n Node) IsNull() bool
- func (n Node) NamedChild(index uint32) Node
- func (n Node) NamedChildCount() uint32
- func (n Node) NamedChildren() []Node
- func (n Node) NamedDescendantForByteRange(start, end uint32) Node
- func (n Node) NamedDescendantForPointRange(start, end Point) Node
- func (n Node) NextNamedSibling() Node
- func (n Node) NextParseState() StateID
- func (n Node) NextSibling() Node
- func (n Node) Parent() Node
- func (n Node) ParseState() StateID
- func (n Node) PrevNamedSibling() Node
- func (n Node) PrevSibling() Node
- func (n Node) StartByte() uint32
- func (n Node) StartPoint() Point
- func (n Node) String() string
- func (n Node) Symbol() Symbol
- func (n Node) Tree() *Tree
- func (n Node) Type() string
- func (n Node) Walk() *TreeCursor
- type ParseAction
- type ParseActionEntry
- type Parser
- func (p *Parser) IncludedRanges() []Range
- func (p *Parser) Language() *Language
- func (p *Parser) Parse(oldTree *Tree, input Input) *Tree
- func (p *Parser) ParseString(oldTree *Tree, source []byte) *Tree
- func (p *Parser) Reset()
- func (p *Parser) SetIncludedRanges(ranges []Range) bool
- func (p *Parser) SetLanguage(language *Language) bool
- type Point
- type Range
- type StateID
- type Symbol
- type SymbolMetadata
- type Tables
- type Tree
- func (t *Tree) Copy() *Tree
- func (t *Tree) Edit(edit InputEdit)
- func (t *Tree) IncludedRanges() []Range
- func (t *Tree) Language() *Language
- func (t *Tree) RootNode() Node
- func (t *Tree) RootNodeWithOffset(offsetBytes uint32, offsetExtent Point) Node
- func (t *Tree) String() string
- func (t *Tree) Walk() *TreeCursor
- type TreeCursor
- func (c *TreeCursor) Copy() *TreeCursor
- func (c *TreeCursor) Depth() uint32
- func (c *TreeCursor) FieldName() string
- func (c *TreeCursor) GotoFirstChild() bool
- func (c *TreeCursor) GotoNextSibling() bool
- func (c *TreeCursor) GotoParent() bool
- func (c *TreeCursor) GotoPreviousSibling() bool
- func (c *TreeCursor) Node() Node
- func (c *TreeCursor) Reset(n Node)
Constants ¶
const ( ParseActionTypeShift uint8 = iota ParseActionTypeReduce ParseActionTypeAccept ParseActionTypeRecover )
Parse action types.
const ( MinABIVersion = 13 MaxABIVersion = 15 )
A generator must refuse an ABI outside this range: the tables would parse nothing. See checkABI in cmd/ts-translate.
const ( BinOr int32 = iota BinAnd BinBitOr BinXor BinBitAnd BinEq BinNe BinLt BinGt BinLe BinGe BinShl BinShr BinAdd BinSub BinMul BinDiv BinMod )
The binary operators a lex condition uses, as OpBinary's Arg.
Variables ¶
This section is empty.
Functions ¶
func DecodeUTF8 ¶
DecodeUTF8 mirrors the reference implementation's byte sequence handling, including how far it advances over an ill formed sequence.
func DecodeUTF16BE ¶
DecodeUTF16BE decodes a single code point from big endian input.
func DecodeUTF16LE ¶
DecodeUTF16LE decodes a single code point from little endian input.
func EncodeTables ¶
EncodeTables writes a grammar's data as a compressed blob.
func SetContains ¶
func SetContains(ranges []CharacterRange, lookahead int32) bool
SetContains reports whether a code point falls in any of the sorted ranges.
Types ¶
type CharacterRange ¶
CharacterRange is an inclusive range of code points.
type ExternalScanner ¶
type ExternalScanner interface {
Create() any
Destroy(payload any)
Scan(payload any, lexer *Lexer, validSymbols []bool) bool
Serialize(payload any, buffer []byte) uint32
Deserialize(payload any, buffer []byte)
}
ExternalScanner is the set of callbacks a hand written scanner provides.
type FieldMapEntry ¶
FieldMapEntry associates a field name with a child position.
type Input ¶
type Input struct {
Read func(byteOffset uint32, position Point) []byte
Encoding InputEncoding
Decode func([]byte) (int32, uint32)
}
Input supplies source text to the parser in chunks.
type InputEdit ¶
type InputEdit struct {
StartByte uint32
OldEndByte uint32
NewEndByte uint32
StartPoint Point
OldEndPoint Point
NewEndPoint Point
}
InputEdit describes a change to a document.
type InputEncoding ¶
type InputEncoding int
InputEncoding selects how the source bytes are decoded.
const ( EncodingUTF8 InputEncoding = iota EncodingUTF16LE EncodingUTF16BE EncodingCustom )
Supported source encodings.
type Language ¶
type Language struct {
ABIVersion uint32
SymbolCount uint32
AliasCount uint32
TokenCount uint32
ExternalTokenCount uint32
StateCount uint32
LargeStateCount uint32
ProductionIDCount uint32
FieldCount uint32
MaxAliasSequenceLen uint16
ParseTable []uint16
SmallParseTable []uint16
SmallParseTableMap []uint32
ParseActions []ParseActionEntry
SymbolNames []string
FieldNames []string
FieldMapSlices []MapSlice
FieldMapEntries []FieldMapEntry
SymbolMetadataTable []SymbolMetadata
PublicSymbolMap []Symbol
AliasMap []uint16
AliasSequences []Symbol
LexModes []LexerMode
LexFn func(*Lexer, StateID) bool
KeywordLexFn func(*Lexer, StateID) bool
KeywordCaptureToken Symbol
ExternalScannerStates []bool
ExternalScannerSymbol []Symbol
Scanner ExternalScanner
PrimaryStateIDs []StateID
Name string
ReservedWords []Symbol
MaxReservedWordSetLen uint16
SupertypeCount uint32
SupertypeSymbols []Symbol
SupertypeMapSlices []MapSlice
SupertypeMapEntries []Symbol
Metadata LanguageMetadata
}
Language holds a grammar's parse tables and lexer functions.
func LoadGrammar ¶
func LoadGrammar(name string, blob []byte, scanner ExternalScanner) *Language
LoadGrammar decodes a grammar's table blob and wires its lexer. Pass a nil scanner for a grammar with no external tokens.
It panics on a blob this runtime cannot read. A grammar that parses nothing is worse to hand back than a stop at the point the fault is visible.
func (*Language) AliasesForSymbol ¶
AliasesForSymbol reports every public name a symbol can take.
func (*Language) FieldIDForName ¶
FieldIDForName returns the identifier of a named field.
func (*Language) FieldNameForID ¶
FieldNameForID returns a field's name.
func (*Language) LanguageName ¶
LanguageName reports the grammar's own name.
func (*Language) StateIsPrimary ¶
StateIsPrimary reports whether a state is the canonical any of its class.
func (*Language) Supertypes ¶
Supertypes reports the grammar's supertype symbols.
func (*Language) SymbolCountTotal ¶
SymbolCountTotal reports the number of symbols including aliases.
func (*Language) SymbolForName ¶
SymbolForName finds a symbol by its display name.
func (*Language) SymbolName ¶
SymbolName returns the display name for a symbol.
type LanguageMetadata ¶
LanguageMetadata is a grammar's semantic version.
type LexOp ¶
type LexOp uint8
LexOp is a single instruction of a lexer program.
const ( // OpPushLit pushes Arg. OpPushLit LexOp = iota // OpPushLookahead pushes the code point at the current position. OpPushLookahead // OpPushEOF pushes the end-of-input flag this iteration read. OpPushEOF // OpPushSet pushes whether character set Arg holds the lookahead. OpPushSet // OpNot and OpNeg replace the top of the stack. OpNot OpNeg // OpBinary combines the top values with binary operator Arg. OpBinary // OpJumpIfFalse pops, and jumps to Arg when the value is empty. OpJumpIfFalse // OpDispatch jumps to the body of the case for the current state. OpDispatch // OpAdvance and OpSkip move to state Arg. Skip drops what it consumed. OpAdvance OpSkip // OpAdvanceMap reads map Arg, and advances on a lookahead the map names. OpAdvanceMap // OpAccept marks the token Arg, ending here. OpAccept // OpEnd answers with what the run accepted. OpReturn answers Arg. OpEnd OpReturn // OpReadEOF reads the end-of-input flag. OpReadEOF )
type LexProgram ¶
LexProgram is a lexer state machine. Code opens with the prologue and an OpDispatch, and the case bodies follow it.
func (*LexProgram) Run ¶
func (p *LexProgram) Run(sets [][]CharacterRange, lexer *Lexer, state StateID) bool
Run walks the program for a single token.
type Lexer ¶
type Lexer struct {
// Lookahead is the code point at the current position.
Lookahead int32
// ResultSymbol is set by a lex function when it accepts a token.
ResultSymbol Symbol
// contains filtered or unexported fields
}
Lexer scans the source text on behalf of a grammar's lex functions.
func (*Lexer) IsAtIncludedRangeStart ¶
IsAtIncludedRangeStart reports whether the lexer sits on a range boundary.
type Node ¶
type Node struct {
// contains filtered or unexported fields
}
Node is a single node in a syntax tree.
func (Node) ChildByFieldID ¶
ChildByFieldID returns the child registered under the given field.
func (Node) ChildByFieldName ¶
ChildByFieldName returns the child registered under the given field name.
func (Node) ChildCount ¶
ChildCount reports the number of visible children.
func (Node) ChildWithDescendant ¶
ChildWithDescendant returns the child that contains the given descendant.
func (Node) Children ¶
Children returns every visible child, as Child returns each. It reads the children once.
func (Node) DescendantCount ¶
DescendantCount reports how many visible nodes the subtree holds.
func (Node) DescendantForByteRange ¶
DescendantForByteRange returns the smallest node spanning the byte range.
func (Node) DescendantForPointRange ¶
DescendantForPointRange returns the smallest node spanning the point range.
func (Node) FieldNameForChild ¶
FieldNameForChild reports the field name of the visible child at an index.
func (Node) FieldNameForNamedChild ¶
FieldNameForNamedChild reports the field name of the named child at an index.
func (Node) GrammarSymbol ¶
GrammarSymbol reports the node's symbol before alias resolution.
func (Node) GrammarType ¶
GrammarType reports the node's rule name before alias resolution.
func (Node) HasChanges ¶
HasChanges reports whether an edit touched the node.
func (Node) NamedChild ¶
NamedChild returns the named child at the given index.
func (Node) NamedChildCount ¶
NamedChildCount reports the number of named children.
func (Node) NamedChildren ¶
NamedChildren returns every named child, as NamedChild returns each.
func (Node) NamedDescendantForByteRange ¶
NamedDescendantForByteRange returns the smallest named node spanning the range.
func (Node) NamedDescendantForPointRange ¶
NamedDescendantForPointRange returns the smallest named node spanning the range.
func (Node) NextNamedSibling ¶
NextNamedSibling returns the next named node with the same parent.
func (Node) NextParseState ¶
NextParseState reports the state reached after the node.
func (Node) NextSibling ¶
NextSibling returns the next visible node with the same parent.
func (Node) ParseState ¶
ParseState reports the parse state the node was created in.
func (Node) PrevNamedSibling ¶
PrevNamedSibling returns the named node with the same parent.
func (Node) PrevSibling ¶
PrevSibling returns the visible node with the same parent.
func (Node) StartPoint ¶
StartPoint reports the node's earliest row and column.
type ParseAction ¶
type ParseAction struct {
Type uint8
State StateID
Extra bool
Repetition bool
ChildCount uint8
Symbol Symbol
DynamicPrecedence int16
ProductionID uint16
}
ParseAction is a single entry of a parse table cell.
type ParseActionEntry ¶
type ParseActionEntry struct {
Action ParseAction
Count uint8
Reusable bool
}
ParseActionEntry is either a group header or a single action.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser turns source text into a syntax tree.
func (*Parser) IncludedRanges ¶
IncludedRanges reports the regions the parser reads.
func (*Parser) ParseString ¶
ParseString builds a syntax tree from a source string.
func (*Parser) Reset ¶
func (p *Parser) Reset()
Reset clears any parse state left over from a previous call.
func (*Parser) SetIncludedRanges ¶
SetIncludedRanges restricts parsing to the given regions.
func (*Parser) SetLanguage ¶
SetLanguage assigns the grammar this parser uses.
type Symbol ¶
type Symbol = uint16
Symbol identifies a terminal or non-terminal in a grammar.
const BuiltinSymEnd Symbol = 0
BuiltinSymEnd is the symbol for the end of input.
const BuiltinSymError Symbol = 0xFFFF
BuiltinSymError is the symbol for an error node.
type SymbolMetadata ¶
SymbolMetadata records how a symbol appears in a tree.
type Tables ¶
type Tables struct {
Language
// CharacterSets holds the sets the lexer searches.
CharacterSets [][]CharacterRange
// Lex and KeywordLex are the grammar's lexer state machines.
Lex *LexProgram
KeywordLex *LexProgram
}
The embedded Language's function fields and its scanner are not encoded.
func DecodeTables ¶
DecodeTables reads back what EncodeTables wrote.
type Tree ¶
type Tree struct {
// contains filtered or unexported fields
}
Tree is a syntax tree produced by a parser.
func (*Tree) IncludedRanges ¶
IncludedRanges reports the regions the parser read.
func (*Tree) RootNodeWithOffset ¶
RootNodeWithOffset returns the root shifted by the given offset.
func (*Tree) Walk ¶
func (t *Tree) Walk() *TreeCursor
Walk returns a cursor positioned on the tree's root.
type TreeCursor ¶
type TreeCursor struct {
// contains filtered or unexported fields
}
TreeCursor walks a syntax tree. A step to a sibling costs the same at any index, because each level reads its children once.
func (*TreeCursor) Copy ¶
func (c *TreeCursor) Copy() *TreeCursor
Copy returns an independent cursor at the same position.
func (*TreeCursor) Depth ¶
func (c *TreeCursor) Depth() uint32
Depth reports how far the cursor has descended from its starting node.
func (*TreeCursor) FieldName ¶
func (c *TreeCursor) FieldName() string
FieldName reports the field the current node fills in its parent.
func (*TreeCursor) GotoFirstChild ¶
func (c *TreeCursor) GotoFirstChild() bool
GotoFirstChild moves to the earliest visible child, reporting whether a single exists.
func (*TreeCursor) GotoNextSibling ¶
func (c *TreeCursor) GotoNextSibling() bool
GotoNextSibling moves to the next visible sibling, reporting whether a single exists.
func (*TreeCursor) GotoParent ¶
func (c *TreeCursor) GotoParent() bool
GotoParent moves to the parent node, reporting whether a single exists.
func (*TreeCursor) GotoPreviousSibling ¶
func (c *TreeCursor) GotoPreviousSibling() bool
GotoPreviousSibling moves to the visible sibling.
func (*TreeCursor) Node ¶
func (c *TreeCursor) Node() Node
Node reports the node the cursor sits on.
func (*TreeCursor) Reset ¶
func (c *TreeCursor) Reset(n Node)
Reset moves the cursor to the given node and forgets its history.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
ts-fetch
command
Command ts-fetch puts a grammar's upstream sources where ts-translate can read them.
|
Command ts-fetch puts a grammar's upstream sources where ts-translate can read them. |
|
ts-goldens
command
Command ts-goldens records the reference parse of a source file.
|
Command ts-goldens records the reference parse of a source file. |
|
ts-translate
command
Command ts-translate turns a tree-sitter generated parser.c into the table blob a grammar package embeds, and the loader that embeds it.
|
Command ts-translate turns a tree-sitter generated parser.c into the table blob a grammar package embeds, and the loader that embeds it. |
|
grammars
|
|
|
clang
Package clang holds the tree-sitter grammar for the C language.
|
Package clang holds the tree-sitter grammar for the C language. |
|
golang
Package golang holds the tree-sitter grammar for the Go language.
|
Package golang holds the tree-sitter grammar for the Go language. |
|
tsx
Package tsx is TypeScript with JSX.
|
Package tsx is TypeScript with JSX. |
|
internal
|
|
|
corpus
Package corpus reads the test corpus format that every tree-sitter grammar repository ships under test/corpus.
|
Package corpus reads the test corpus format that every tree-sitter grammar repository ships under test/corpus. |
|
goldens
Package goldens compares parse results against files produced by the reference tree-sitter command line tool.
|
Package goldens compares parse results against files produced by the reference tree-sitter command line tool. |
|
grammarsrc
Package grammarsrc puts a grammar's upstream sources where a translator can read them.
|
Package grammarsrc puts a grammar's upstream sources where a translator can read them. |