generate

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package generate is the parser generator of transit, a port of the crate crates/generate of upstream tree-sitter (D7). It reads a grammar from grammar.json (D17), builds the parse table and the lexer tables, and gives them to a backend, which writes a parser (D8).

ParserForGrammar runs the whole generator on a grammar.json, and ParserForGrammarWithOpts gives the ABI version and the other options. The package writes no parser itself. A backend, such as the C backend in generate/backend/c, takes a RenderInput and writes one.

Index

Constants

View Source 
const (
	ABIVersionMin = 14
	ABIVersionMax = LanguageVersion
)

The lowest and the highest ABI version that a backend writes.

ABIVersionMin and ABIVersionMax are ABI_VERSION_MIN and ABI_VERSION_MAX of render.rs, which generate.rs exports.

View Source 
const LanguageVersion = 15

LanguageVersion is the ABI version that the generator writes when no other one is asked for.

LanguageVersion is LANGUAGE_VERSION.

View Source 
const LargeCharacterRangeCount = 8

LargeCharacterRangeCount is the number of ranges above which a set of characters is a large character set, which the parser holds in a table of its own.

LargeCharacterRangeCount is LARGE_CHARACTER_RANGE_COUNT.

View Source 
const NoReservedWords uint16 = 1<<16 - 1

NoReservedWords is the value of ProductionStep.Reserved that means no set of reserved words at all. Only the start production that the generator adds has it, and it never indexes the table of reserved word sets.

NoReservedWords is ProductionStep::NO_RESERVED_WORDS.

View Source 
const StartProductionID uint32 = math.MaxUint32

StartProductionID is the production id of the start production, which the generator adds and which is not in the grammar.

StartProductionID is START_PRODUCTION_ID.

Variables

View Source 
var (
	//go:embed templates/alloc.h
	AllocHeader string
	//go:embed templates/array.h
	ArrayHeader string
	//go:embed templates/parser.h
	ParserHeader string
)

The headers that a generated parser includes, copied from upstream into generate/templates. ParserInDirectory writes them to src/tree_sitter.

AllocHeader, ArrayHeader and ParserHeader are ALLOC_HEADER, ARRAY_HEADER and PARSER_HEADER.

View Source 
var (
	// SymbolEndValue is the symbol at the end of the input.
	SymbolEndValue = Symbol{/* contains filtered or unexported fields */}
	// SymbolEndOfNonTerminalExtraValue ends a non-terminal extra.
	SymbolEndOfNonTerminalExtraValue = Symbol{/* contains filtered or unexported fields */}
)

The symbols with no index.

Functions

func BuildParseTable 

func BuildParseTable(
	syntaxGrammar *SyntaxGrammar,
	lexicalGrammar *LexicalGrammar,
	itemSetBuilder *ParseItemSetBuilder,
	variableInfo []VariableInfo,
	strPool *StrPool,
	diagnostics *[]Diagnostic,
) (ParseTable, *ParseStateInfo, error)

BuildParseTable builds the parse table of a grammar, and returns it with the information about its states. It adds a diagnostic for the expected conflicts that the grammar does not need. The error is a *ParseTableBuilderError.

BuildParseTable is build_parse_table.

func CompareAdvanceAction 

func CompareAdvanceAction(a, b AdvanceAction) int

CompareAdvanceAction orders two actions by state, then by InMainToken.

CompareAdvanceAction is the Ord of AdvanceAction.

func CompareChildType 

func CompareChildType(a, b ChildType) int

CompareChildType orders child types as the derived Ord of upstream does: a symbol before an alias, then symbols by CompareSymbol, and aliases by their value and then by whether they are named.

CompareChildType is the Ord of ChildType.

func CompareLexState 

func CompareLexState(a, b *LexState) int

CompareLexState orders two states by the accept action, then by the action at the end of the input, then by the advance actions. None comes before a value.

CompareLexState is the Ord of LexState.

func CompareSymbol 

func CompareSymbol(a, b Symbol) int

CompareSymbol orders symbols by kind, then by index, as the Ord of Symbol does.

func MinimizeParseTable 

func MinimizeParseTable(
	parseTable *ParseTable,
	syntaxGrammar *SyntaxGrammar,
	lexicalGrammar *LexicalGrammar,
	simpleAliases AliasMap,
	tokenConflictMap *TokenConflictMap,
	keywords *TokenSet,
	strPool *StrPool,
	optimizations OptLevel,
)

MinimizeParseTable makes the parse table smaller. When optimizations holds OptLevelMergeStates, it merges the states that are compatible. Then it skips the states that only reduce a unit rule, removes the states that no state refers to, and orders the states by descending size. tokenConflictMap and keywords are used only to merge states.

MinimizeParseTable is minimize_parse_table.

func NodeTypesJSON 

func NodeTypesJSON(syntaxGrammar *SyntaxGrammar, lexicalGrammar *LexicalGrammar, defaultAliases AliasMap, variableInfo []VariableInfo, strPool *StrPool) (string, error)

NodeTypesJSON returns the text of node-types.json, as upstream writes it with serde_json::to_string_pretty. The text has no newline at its end, because the text of upstream has none.

The error is a *SuperTypeCycleError.

NodeTypesJSON is generate_node_types_json.

func ParserForGrammar 

func ParserForGrammar(grammarJSON []byte, semanticVersion *SemanticVersion, optimizations OptLevel, backend Backend, diagnostics *[]Diagnostic) (string, string, error)

ParserForGrammar runs the generator on a grammar.json at LanguageVersion, and returns the name of the grammar and the code that the backend writes.

ParserForGrammar is generate_parser_for_grammar, without the prefix Generate, which would repeat the name of the package.

func ParserInDirectory 

func ParserInDirectory(repoPath, outPath, grammarPath string, abiVersion int, generateParser bool, optimizations OptLevel, backend Backend, diagnostics *[]Diagnostic) error

ParserInDirectory generates the parser of a grammar folder: it reads src/grammar.json of repoPath, or the grammar file of grammarPath, and writes parser.c, node-types.json and the headers to outPath, or to the folder src of the grammar. With generateParser false, it writes node-types.json only. repoPath is the folder of the grammar when grammarPath is empty.

ParserInDirectory is generate_parser_in_directory, without the prefix Generate, which would repeat the name of the package. It reads no grammar.js (D17).

func PrecDisplay 

func PrecDisplay(prec Precedence, strPool *StrPool) string

PrecDisplay returns the text of a precedence: none, a number, or a name in quotes.

PrecDisplay is prec_display.

func PreferToken 

func PreferToken(grammar *LexicalGrammar, leftPrecedence int32, leftID int, rightPrecedence int32, rightID int) bool

PreferToken reports whether the lexer prefers the left token over the right token, when both match the same string. A token is its precedence and its variable index.

PreferToken is TokenConflictMap::prefer_token. Upstream takes each token as a tuple (i32, usize).

func PreferTransition 

func PreferTransition(grammar *LexicalGrammar, t *NfaTransition, completedID int, completedPrecedence int32, hasSeparatorTransitions bool) bool

PreferTransition reports whether the lexer prefers to take transition t over accepting the completed token.

PreferTransition is TokenConflictMap::prefer_transition.

func SplitStateIDGroups 

func SplitStateIDGroups[S any](
	states []S,
	stateIDsByGroupID *[][]uint32,
	groupIDsByStateID []uint32,
	startGroupID uint32,
	criterion SplitCriterion[S],
) bool

SplitStateIDGroups splits every group from startGroupID on, including the groups split off from them, so that no group has two states that criterion separates. It reports whether any group split.

A group's states are scanned in order: each state stays unless it must be split from a state that stayed before it, and the states that don't stay form a new group, ordered by the state they were split from, then by position.

SplitStateIDGroups is split_state_id_groups. Upstream holds each element of split_from as an Option<u32>, and the Go form holds the position, or -1 for None.

Types

type ActionList 

type ActionList []ParseAction

ActionList is the list of actions of one entry of the parse table.

ActionList is ActionList. Upstream holds a list of one action inline, and its slice is the same as the Go slice. Two lists are equal when their actions are equal, so they compare with slices.Equal. A copy of an ActionList shares its memory, so a list that is copied and then changed must be copied with Clone.

func ActionListFromSlice 

func ActionListFromSlice(actions []ParseAction) ActionList

ActionListFromSlice returns a list that holds a copy of the actions.

ActionListFromSlice is ActionList::from_slice.

func (*ActionList) Clear 

func (l *ActionList) Clear()

Clear removes every action.

Clear is ActionList::clear.

func (ActionList) Clone 

func (l ActionList) Clone() ActionList

Clone returns a copy of the list.

Clone is the Clone of ActionList.

func (*ActionList) KeepLast 

func (l *ActionList) KeepLast()

KeepLast removes every action but the last one.

KeepLast is ActionList::keep_last.

func (*ActionList) Pop 

func (l *ActionList) Pop() (ParseAction, bool)

Pop removes the last action and returns it, and false when the list is empty.

Pop is ActionList::pop.

func (*ActionList) Push 

func (l *ActionList) Push(action ParseAction)

Push adds an action at the end of the list.

Push is ActionList::push.

type ActionListID 

type ActionListID uint32

ActionListID is the index of an action list in an ActionListPool. Its high bit tells whether the list is reusable.

ActionListID is ActionListId.

func NewActionListID 

func NewActionListID(index uint32, reusable bool) ActionListID

NewActionListID returns the id of the list at an index.

NewActionListID is ActionListId::new.

func (ActionListID) Index 

func (id ActionListID) Index() int

Index returns the index of the list in its pool.

Index is ActionListId::index.

func (ActionListID) Reusable 

func (id ActionListID) Reusable() bool

Reusable reports whether the list is reusable.

Reusable is ActionListId::reusable.

func (*ActionListID) SetReusable 

func (id *ActionListID) SetReusable(reusable bool)

SetReusable sets whether the list is reusable.

SetReusable is ActionListId::set_reusable.

type ActionListPool 

type ActionListPool struct {
	// contains filtered or unexported fields
}

ActionListPool holds each action list of the terminal entries of the parse table once.

Every entry of a state and a terminal has an action list, but in a grammar 98 to 99 percent of these lists are the same as another list. Each entry holds an ActionListID. The id is the index of a range, and the range is the slice of the actions of one list.

The builder of the parse table interns the lists of each state into this pool as it completes the state, and every list becomes an id. Then minimize changes the targets of the shift actions in the pool. Canonicalize builds the pool again before render, with each list stored once. The list at index 0 of the pool is always the empty list.

ActionListPool is ActionListPool. The zero ActionListPool is empty and ready to use.

func (*ActionListPool) Canonicalize 

func (p *ActionListPool) Canonicalize(states []ParseState)

Canonicalize builds the pool again, with only the lists that the states use, each stored once, and changes the ids of the states to the new indices. The empty list gets index 0.

Canonicalize is ActionListPool::canonicalize.

func (*ActionListPool) Get 

func (p *ActionListPool) Get(id ActionListID) []ParseAction

Get returns the actions of a list.

Get is ActionListPool::get.

func (*ActionListPool) Intern 

func (p *ActionListPool) Intern(dedup map[string]uint32, list ActionList) uint32

Intern returns the index of a list. When dedup does not hold the list yet, Intern adds the list to the pool and to dedup. Make dedup with make(map[string]uint32).

Intern is ActionListPool::intern. Upstream keys dedup with the list, and the Go key is the string of actionListKey. Upstream only looks up dedup.

func (*ActionListPool) Len 

func (p *ActionListPool) Len() int

Len returns the number of lists in the pool.

Len is ActionListPool::len.

func (*ActionListPool) Push 

func (p *ActionListPool) Push(list []ParseAction) uint32

Push adds a list to the pool and returns its index.

Push is ActionListPool::push.

type AdvanceAction 

type AdvanceAction struct {
	State       LexStateID
	InMainToken bool
}

AdvanceAction is the action of the lexer that moves to a state.

AdvanceAction is AdvanceAction.

type Alias 

type Alias struct {
	Value   StrID
	IsNamed bool
}

Alias is an alias of a rule: the name that it shows, and whether the node is named.

Alias is Alias.

func (Alias) Kind 

func (a Alias) Kind() VariableType

Kind returns the variable type that the alias gives.

Kind is Alias::kind.

type AliasMap 

type AliasMap map[Symbol]Alias

AliasMap maps a symbol to its alias.

AliasMap is AliasMap, a BTreeMap upstream, which iterates in the order of the symbols. Sorted gives that order.

func (AliasMap) Sorted 

func (m AliasMap) Sorted() iter.Seq2[Symbol, Alias]

Sorted returns each symbol and its alias, in the order of the symbols.

type AmbiguousExtraError 

type AmbiguousExtraError struct {
	ParentSymbols []string
}

AmbiguousExtraError is a non-terminal extra whose end is ambiguous. It holds the names of the rules that conflict.

AmbiguousExtraError is AmbiguousExtraError.

func (*AmbiguousExtraError) Error 

func (e *AmbiguousExtraError) Error() string

Error returns the names of the rules, separated by commas.

Error is the Display of AmbiguousExtraError.

type Associativity 

type Associativity uint8

Associativity is the associativity of a precedence.

Associativity is Associativity. Upstream wraps it in an Option, and AssociativityNone stands for the None of that Option. 

const (
	AssociativityNone Associativity = iota
	AssociativityLeft
	AssociativityRight
)

The associativities.

type Backend 

type Backend interface {
	// Render writes the parser of a grammar.
	Render(in *RenderInput) (string, error)
}

Backend writes a parser in one target language (D8).

type BitVec 

type BitVec struct {
	// contains filtered or unexported fields
}

BitVec is a vector of bits in words of 64 bits. Token sets are joined many times, and a join of whole words is faster than a join of bits.

The bits at and after Len are always zero.

BitVec is BitVec.

func NewBitVecWithCapacity 

func NewBitVecWithCapacity(n int) BitVec

NewBitVecWithCapacity returns an empty BitVec with room for n bits.

NewBitVecWithCapacity is BitVec::with_capacity.

func (*BitVec) Clone 

func (b *BitVec) Clone() BitVec

Clone returns a copy of the vector.

Clone is the Clone of BitVec.

func (*BitVec) Compare 

func (b *BitVec) Compare(other *BitVec) int

Compare orders two vectors. At the lowest bit where they differ, the vector with the bit set is the greater.

Compare is the Ord of BitVec.

func (*BitVec) Equal 

func (b *BitVec) Equal(other *BitVec) bool

Equal reports whether two vectors hold the same bits. A missing word counts as zero, so two vectors that differ only in zero words at the end are equal.

Equal is the PartialEq of BitVec.

func (*BitVec) Get 

func (b *BitVec) Get(index int) (bool, bool)

Get returns the bit at an index, and false when the index is not before Len.

Get is BitVec::get.

func (*BitVec) InsertAll 

func (b *BitVec) InsertAll(other *BitVec) bool

InsertAll sets each bit that is set in other, word by word, and reports whether any bit was new.

InsertAll is BitVec::insert_all.

func (*BitVec) Last 

func (b *BitVec) Last() (bool, bool)

Last returns the last bit, and false when the vector is empty.

Last is BitVec::last.

func (*BitVec) Len 

func (b *BitVec) Len() int

Len returns the number of bits.

Len is BitVec::len.

func (*BitVec) Pop 

func (b *BitVec) Pop() (bool, bool)

Pop removes the last bit and returns it, and false when the vector is empty.

Pop is BitVec::pop.

func (*BitVec) Resize 

func (b *BitVec) Resize(n int, val bool)

Resize changes Len to n. A new bit takes the value val.

Resize is BitVec::resize.

func (*BitVec) Set 

func (b *BitVec) Set(index int, val bool)

Set sets the bit at an index before Len.

Set is BitVec::set.

func (*BitVec) UnsetAll added in v0.4.0 

func (b *BitVec) UnsetAll()

UnsetAll unsets every bit, keeping the length.

UnsetAll is BitVec::unset_all.

func (*BitVec) Words 

func (b *BitVec) Words() []uint64

Words returns the words in use.

Words is BitVec::as_slice.

type CharacterSet 

type CharacterSet struct {
	// contains filtered or unexported fields
}

CharacterSet is a set of characters, as sorted ranges that do not touch. The zero CharacterSet is the empty set.

CharacterSet is CharacterSet, and the zero CharacterSet is CharacterSet::empty. A method that takes self in upstream, and returns Self, works like append: it can change the set that it is called on, so use only the set that it returns.

func CharacterSetFromChar 

func CharacterSetFromChar(c rune) CharacterSet

CharacterSetFromChar returns the set of one character.

CharacterSetFromChar is CharacterSet::from_char.

func (CharacterSet) Add 

func (s CharacterSet) Add(other CharacterSet) CharacterSet

Add returns s with the characters of other added.

Add is CharacterSet::add.

func (CharacterSet) AddChar 

func (s CharacterSet) AddChar(c rune) CharacterSet

AddChar returns s with a character added.

AddChar is CharacterSet::add_char.

func (CharacterSet) AddRange 

func (s CharacterSet) AddRange(start, end rune) CharacterSet

AddRange returns s with the characters from start to end added. The range holds end.

AddRange is CharacterSet::add_range.

func (*CharacterSet) Assign 

func (s *CharacterSet) Assign(other CharacterSet)

Assign makes s hold the characters of other. It reuses the memory of s.

Assign is CharacterSet::assign.

func (CharacterSet) CharCodes 

func (s CharacterSet) CharCodes() iter.Seq[uint32]

CharCodes returns each code point in the set, surrogates included.

CharCodes is CharacterSet::char_codes.

func (CharacterSet) Chars 

func (s CharacterSet) Chars() iter.Seq[rune]

Chars returns each character in the set. It skips the code points that are not characters, which are the surrogates.

Chars is CharacterSet::chars.

func (CharacterSet) Clone 

func (s CharacterSet) Clone() CharacterSet

Clone returns a copy of the set. A method that works like append changes the memory of the set, so a caller that uses a set again copies it first.

Clone is the Clone of CharacterSet.

func (CharacterSet) Compare 

func (s CharacterSet) Compare(other CharacterSet) int

Compare orders two sets: by the number of code points, then range by range, by the length of the range and then by its first code point.

Compare is the Ord of CharacterSet. Upstream compares two ranges of the same length code point by code point, and the first code points decide that.

func (CharacterSet) Contains 

func (s CharacterSet) Contains(c rune) bool

Contains reports whether the set holds a character.

Contains is CharacterSet::contains.

func (CharacterSet) Difference 

func (s CharacterSet) Difference(other CharacterSet) CharacterSet

Difference returns the characters of s that are not in other. Like Negate, it can change s, and it can change other too.

Difference is CharacterSet::difference.

func (CharacterSet) DoesIntersect 

func (s CharacterSet) DoesIntersect(other CharacterSet) bool

DoesIntersect reports whether s and other have a character in common.

DoesIntersect is CharacterSet::does_intersect.

func (CharacterSet) Equal 

func (s CharacterSet) Equal(other CharacterSet) bool

Equal reports whether two sets hold the same ranges.

Equal is the PartialEq of CharacterSet.

func (CharacterSet) IsEmpty 

func (s CharacterSet) IsEmpty() bool

IsEmpty reports whether the set is empty.

IsEmpty is CharacterSet::is_empty.

func (CharacterSet) Negate 

func (s CharacterSet) Negate() CharacterSet

Negate returns the set of the characters that are not in s.

Negate is CharacterSet::negate.

func (CharacterSet) RangeCount 

func (s CharacterSet) RangeCount() int

RangeCount returns the number of ranges in the set.

RangeCount is CharacterSet::range_count.

func (CharacterSet) Ranges 

func (s CharacterSet) Ranges() iter.Seq2[rune, rune]

Ranges returns each range of the set as its first and its last character. A range starts and ends at a character, not at a surrogate, and a range of surrogates only is skipped.

Ranges is CharacterSet::ranges.

func (*CharacterSet) RemoveIntersection 

func (s *CharacterSet) RemoveIntersection(other *CharacterSet) CharacterSet

RemoveIntersection returns the characters that are in both s and other, and removes them from both.

RemoveIntersection is CharacterSet::remove_intersection.

func (CharacterSet) SimplifyIgnoring 

func (s CharacterSet) SimplifyIgnoring(ruledOut CharacterSet) CharacterSet

SimplifyIgnoring returns a set with fewer ranges, which holds the same characters as s where the characters in ruledOut do not matter.

SimplifyIgnoring is CharacterSet::simplify_ignoring.

type ChildQuantity 

type ChildQuantity struct {
	// contains filtered or unexported fields
}

ChildQuantity says whether a child exists, whether it is required, and whether it can repeat.

The zero ChildQuantity is ChildQuantity::zero of upstream. The default of upstream is ChildQuantity::one.

ChildQuantity is ChildQuantity.

type ChildType 

type ChildType struct {
	Kind ChildTypeKind
	// Symbol is the symbol of a ChildTypeNormal.
	Symbol Symbol
	// Alias is the alias of a ChildTypeAliased.
	Alias Alias
}

ChildType is the type of a child of a node: a symbol, or an alias.

Only the field of the kind is set, and the other one is zero, so two child types compare with ==. NormalChildType and AliasedChildType make one.

ChildType is ChildType, an enum with data upstream.

func AliasedChildType 

func AliasedChildType(alias Alias) ChildType

AliasedChildType returns the child type of an alias.

AliasedChildType is ChildType::Aliased.

func NormalChildType 

func NormalChildType(symbol Symbol) ChildType

NormalChildType returns the child type of a symbol.

NormalChildType is ChildType::Normal.

type ChildTypeKind 

type ChildTypeKind uint8

ChildTypeKind is the kind of a ChildType. 

const (
	ChildTypeNormal ChildTypeKind = iota
	ChildTypeAliased
)

The kinds of child type, in the order of upstream. A child type sorts by its kind first.

type CoincidentTokenIndex 

type CoincidentTokenIndex struct {
	// contains filtered or unexported fields
}

CoincidentTokenIndex records which pairs of tokens are valid lookaheads in the same parse state.

CoincidentTokenIndex is CoincidentTokenIndex.

func NewCoincidentTokenIndex 

func NewCoincidentTokenIndex(table *ParseTable, lexicalGrammar *LexicalGrammar, wordToken Symbol, hasWordToken bool) *CoincidentTokenIndex

NewCoincidentTokenIndex returns the index of the tokens that are coincident in the states of a parse table. hasWordToken is false when the grammar has no word token.

NewCoincidentTokenIndex is CoincidentTokenIndex::new. Upstream keeps the recorded sets of terminals in two FxHashSets, and only looks them up. The Go form maps the bytes of each sorted set to true. Upstream sorts with sort_unstable, and two indices that compare equal are the same value, so the order is the same in Go.

func (*CoincidentTokenIndex) AllCoincidentStatesHaveWord 

func (idx *CoincidentTokenIndex) AllCoincidentStatesHaveWord(a, b TerminalIndex) bool

AllCoincidentStatesHaveWord reports whether the word token is a valid lookahead in every parse state where tokens a and b are both valid.

AllCoincidentStatesHaveWord is CoincidentTokenIndex::all_coincident_states_have_word.

func (*CoincidentTokenIndex) Contains 

func (idx *CoincidentTokenIndex) Contains(a, b TerminalIndex) bool

Contains reports whether tokens a and b are both valid in some parse state.

Contains is CoincidentTokenIndex::contains.

type ConflictError 

type ConflictError struct {
	SymbolSequence          []string
	ConflictingLookahead    string
	PossibleInterpretations []Interpretation
	PossibleResolutions     []Resolution
}

ConflictError is a conflict that the grammar does not resolve: the sequence of symbols before the conflict, the lookahead, the ways in which the parser can read the input, and the changes to the grammar that resolve the conflict.

ConflictError is ConflictError.

func (*ConflictError) Error 

func (e *ConflictError) Error() string

Error returns the text of the conflict.

Error is the Display of ConflictError.

type ConflictingPrecedenceOrderingError 

type ConflictingPrecedenceOrderingError struct {
	Precedence1 string
	Precedence2 string
}

ConflictingPrecedenceOrderingError is the error of two precedences that one list puts in one order and another list in the other order.

ConflictingPrecedenceOrderingError is ConflictingPrecedenceOrderingError.

func (*ConflictingPrecedenceOrderingError) Error 

func (e *ConflictingPrecedenceOrderingError) Error() string

Error returns the text of the error.

type Diagnostic 

type Diagnostic struct {
	Kind      DiagnosticKind
	Conflicts [][]string
	Name      string
	Flag      rune
	Pattern   string
}

Diagnostic is a warning that the generator reports and that does not stop it.

Diagnostic is Diagnostic, an enum with data upstream. The fields that a kind uses are:

  • DiagnosticUnnecessaryConflicts: Conflicts.
  • DiagnosticUnaryChoice and DiagnosticUnarySeq: Name, which is empty for a rule with no name.
  • DiagnosticSupertypeInlined: Name.
  • DiagnosticUnsupportedRegexFlag: Flag and Pattern.

func (Diagnostic) String 

func (d Diagnostic) String() string

String returns the text of the diagnostic, as upstream writes it.

String is the Display of Diagnostic.

type DiagnosticKind 

type DiagnosticKind uint8

DiagnosticKind is the kind of a diagnostic. 

const (
	DiagnosticUnnecessaryConflicts DiagnosticKind = iota
	DiagnosticUnaryChoice
	DiagnosticUnarySeq
	DiagnosticUnsupportedRegexFlag
	DiagnosticSupertypeInlined
)

The kinds of diagnostic, in the order of upstream.

type DotKeys 

type DotKeys struct {
	Cmp        uint32
	EqWithSyms uint32
}

DotKeys is the pair of identity keys of one production and one position of the dot.

Cmp is the rank of the content that the Ord of ParseItem compares: the dynamic precedence, RequiresEOFLookahead, the length, the precedence and the associativity at the dot, then the aliases and the fields of the steps before the dot, and the steps after the dot in full. Two ranks are equal exactly when the content is equal, so Cmp is also the class of equality of an item that has no preceding inherited fields. EqWithSyms divides Cmp by the symbols of the steps before the dot, which take part in the equality only when HasPrecedingInheritedFields is true.

DotKeys is DotKeys.

type EmptyStringExtraError added in v0.4.0 

type EmptyStringExtraError struct {
	// Name is the name of the extra.
	Name string
}

EmptyStringExtraError is the error of an extra that can match the empty string.

EmptyStringExtraError is EmptyStringExtraError.

func (*EmptyStringExtraError) Error added in v0.4.0 

func (e *EmptyStringExtraError) Error() string

Error returns the text of the error.

type Error 

type Error struct {
	Kind ErrorKind
	// Path is the file or the folder of the error.
	Path string
	// Text is the text of a ErrorLoadGrammarFile or ErrorParseVersion
	// error that has no Err.
	Text string
	// Err is the error of the operating system, when there is one.
	Err error
}

Error is an error of ParserInDirectory. Its text is the text of upstream, but the text of an error of the operating system is the text of Go.

Error is GenerateError, with LoadGrammarError, ParseVersionError and IoError in it, without the prefix Generate, which would repeat the name of the package.

func (*Error) Error 

func (e *Error) Error() string

Error returns the text of the error.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap returns the error of the operating system.

type ErrorKind 

type ErrorKind uint8

ErrorKind is the kind of an Error. 

const (
	ErrorGrammarPath ErrorKind = iota
	ErrorIO
	ErrorLoadGrammarFile
	ErrorGrammarFileNotFound
	ErrorParseVersion
)

The kinds of error, in the order of upstream. The other variants of GenerateError pass the error of a pass through, so ParserInDirectory returns that error as it is.

type ExpandRegexError 

type ExpandRegexError struct {
	Kind ExpandRegexErrorKind
	// Text is the text of the UTF-8 error, for ExpandRegexUTF8.
	Text string
	// Start and End are the bytes of the class, for
	// ExpandRegexNonASCIIByteClass.
	Start, End byte
}

ExpandRegexError is an error of the HIR of a pattern. Its text is the text of upstream.

ExpandRegexError is ExpandRegexError, with NonAsciiByteClassError in it.

func (*ExpandRegexError) Error 

func (e *ExpandRegexError) Error() string

Error returns the text of the error.

type ExpandRegexErrorKind 

type ExpandRegexErrorKind uint8

ExpandRegexErrorKind is the kind of an error of the HIR of a pattern. 

const (
	ExpandRegexUTF8 ExpandRegexErrorKind = iota
	ExpandRegexAssertion
	ExpandRegexNonASCIIByteClass
)

The kinds of error, in the order of upstream.

type ExpandRepeatsError 

type ExpandRepeatsError struct {
	Rule string
}

ExpandRepeatsError is the error of a repeat that can match the empty string at the end of the input.

ExpandRepeatsError is ExpandRepeatsError.

func (*ExpandRepeatsError) Error 

func (e *ExpandRepeatsError) Error() string

Error returns the text of the error.

type ExpandRuleError 

type ExpandRuleError struct {
	Kind ExpandRuleErrorKind
	// Symbol is the symbol of ExpandRuleUnexpectedSymbol.
	Symbol Symbol
	// Context is the name of the set of reserved words of
	// ExpandRuleUnexpectedReserved.
	Context string
}

ExpandRuleError is an error of a rule of a token. Its text is the text of upstream.

ExpandRuleError is ExpandRuleError.

func (*ExpandRuleError) Error 

func (e *ExpandRuleError) Error() string

Error returns the text of the error.

type ExpandRuleErrorKind 

type ExpandRuleErrorKind uint8

ExpandRuleErrorKind is the kind of an error of a rule of a token. 

const (
	ExpandRuleUnexpectedSymbol ExpandRuleErrorKind = iota
	ExpandRuleUnexpectedReserved
	ExpandRuleUnexpectedEOF
)

The kinds of error, in the order of upstream. The variants Parse and ExpandRegex pass their errors through, so a rule returns a *RegexError or an *ExpandRegexError as it is.

type ExpandTokensError 

type ExpandTokensError struct {
	Kind ExpandTokensErrorKind
	// Rule is the name of the token.
	Rule string
	// Err is the error of the rule of the token, for ExpandTokensProcessing.
	Err error
}

ExpandTokensError is an error of the pass that expands the tokens. Its text is the text of upstream.

ExpandTokensError is ExpandTokensError, with ExpandTokensProcessingError in it. The variant ExpandRule passes its error through, so the pass returns that error as it is.

func (*ExpandTokensError) Error 

func (e *ExpandTokensError) Error() string

Error returns the text of the error.

func (*ExpandTokensError) Unwrap 

func (e *ExpandTokensError) Unwrap() error

Unwrap returns the error of the rule of the token.

type ExpandTokensErrorKind 

type ExpandTokensErrorKind uint8

ExpandTokensErrorKind is the kind of an error of the pass that expands the tokens. 

const (
	ExpandTokensEmptyString ExpandTokensErrorKind = iota
	ExpandTokensProcessing
)

The kinds of error, in the order of upstream.

type ExternalToken 

type ExternalToken struct {
	Name StrID
	Kind VariableType
	// CorrespondingInternalToken is the internal token of the same name,
	// when HasCorrespondingInternalToken is true.
	CorrespondingInternalToken    Symbol
	HasCorrespondingInternalToken bool
}

ExternalToken is a token that the external scanner recognizes.

ExternalToken is ExternalToken.

type ExternalTokenIndex 

type ExternalTokenIndex uint32

ExternalTokenIndex is the index of an external token. It is a type of the macro symbol_index, which gives each kind of symbol its own index.

func (ExternalTokenIndex) Symbol 

func (i ExternalTokenIndex) Symbol() Symbol

Symbol returns the symbol of the index.

type ExtractTokensError 

type ExtractTokensError struct {
	Kind ExtractTokensErrorKind
	// Name is the name of the rule or the token, for every kind but
	// ExtractTokensNonSymbolExternalToken.
	Name string
}

ExtractTokensError is an error of the pass that extracts the tokens. Its text is the text of upstream.

ExtractTokensError is ExtractTokensError.

func (*ExtractTokensError) Error 

func (e *ExtractTokensError) Error() string

Error returns the text of the error.

type ExtractTokensErrorKind 

type ExtractTokensErrorKind uint8

ExtractTokensErrorKind is the kind of an error of the pass that extracts the tokens. 

const (
	ExtractTokensEmptyString ExtractTokensErrorKind = iota
	ExtractTokensSupertypeTerminal
	ExtractTokensExternalTokenNonTerminal
	ExtractTokensNonSymbolExternalToken
	ExtractTokensNonTokenReservedWord
)

The kinds of error, in the order of upstream.

type FieldInfo 

type FieldInfo struct {
	Quantity ChildQuantity
	// Types is sorted by CompareChildType, and it has no duplicates.
	Types []ChildType
}

FieldInfo is the types and the quantity of the children of a field, or of a set of children.

The zero FieldInfo has a zero quantity. The default FieldInfo of upstream has the quantity one, and newFieldInfo gives it.

FieldInfo is FieldInfo.

type FieldLocation 

type FieldLocation struct {
	Index     uint32
	Inherited bool
}

FieldLocation is where a field is in a production: the index of the child, and whether the child only passes on the field of a hidden child.

FieldLocation is FieldLocation.

type FieldMap 

type FieldMap map[StrID][]FieldLocation

FieldMap maps the name of a field to its locations.

FieldMap is BTreeMap<StrId, Vec<FieldLocation>>, which iterates in the order of the names. Sorted gives that order.

func (FieldMap) Sorted 

func (m FieldMap) Sorted() iter.Seq2[StrID, []FieldLocation]

Sorted returns each name and its locations, in the order of the names.

type FlattenGrammarError 

type FlattenGrammarError struct {
	Kind FlattenGrammarErrorKind
	// Name is the name of the rule or of the set of reserved words, for
	// every kind but FlattenGrammarTooManyReservedWordSets.
	Name string
	// Count is the number of sets of reserved words, for
	// FlattenGrammarTooManyReservedWordSets.
	Count int
}

FlattenGrammarError is an error of the pass that flattens the grammar. Its text is the text of upstream.

FlattenGrammarError is FlattenGrammarError.

func (*FlattenGrammarError) Error 

func (e *FlattenGrammarError) Error() string

Error returns the text of the error.

type FlattenGrammarErrorKind 

type FlattenGrammarErrorKind uint8

FlattenGrammarErrorKind is the kind of an error of the pass that flattens the grammar. 

const (
	FlattenGrammarNoReservedWordSet FlattenGrammarErrorKind = iota
	FlattenGrammarTooManyReservedWordSets
	FlattenGrammarEmptyString
	FlattenGrammarRecursiveInline
	FlattenGrammarNoReachableProductions
)

The kinds of error, in the order of upstream.

type GeneratedParser 

type GeneratedParser struct {
	Code          string
	NodeTypesJSON string
}

GeneratedParser is the output of the generator: the code that the backend writes, and node-types.json.

GeneratedParser is GeneratedParser. Its field c_code is Code, because a backend other than the C backend writes it too.

func ParserForGrammarWithOpts 

func ParserForGrammarWithOpts(inputGrammar *InputGrammar, abiVersion int, semanticVersion *SemanticVersion, optimizations OptLevel, backend Backend, diagnostics *[]Diagnostic) (*GeneratedParser, error)

ParserForGrammarWithOpts runs the generator on a parsed grammar at an ABI version, and gives the result to a backend. It changes inputGrammar.

ParserForGrammarWithOpts is generate_parser_for_grammar_with_opts, without the prefix Generate, which would repeat the name of the package. The port leaves out its argument report_symbol_name, which only the log of build_tables reads.

type GotoAction 

type GotoAction struct {
	Kind  GotoActionKind
	State ParseStateID
}

GotoAction is the action of the parser for a non-terminal: go to a state, or shift an extra.

GotoAction is GotoAction, an enum with data upstream. Only GotoActionGoto uses State, and State is zero for GotoActionShiftExtra, so two actions compare with ==.

type GotoActionKind 

type GotoActionKind uint8

GotoActionKind is the kind of a goto action. 

const (
	GotoActionGoto GotoActionKind = iota
	GotoActionShiftExtra
)

The kinds of goto action, in the order of the enum GotoAction.

type IndexMap 

type IndexMap[K comparable, V any] struct {
	// contains filtered or unexported fields
}

IndexMap is a map that keeps its keys in the order in which they were first inserted. The zero IndexMap is empty and ready to use.

An IndexMap must not be copied while it is in use, because a copy shares part of its memory with the original. Copy it with Clone. A pointer that a method returns into the map is valid only until the next insert of a new key.

IndexMap is IndexMap of the indexmap crate. Upstream hashes its keys with FxHasher, which decides only how a key is found, not the order.

func (*IndexMap[K, V]) All 

func (m *IndexMap[K, V]) All() iter.Seq2[K, V]

All returns each key and value, in order.

All is IndexMap::iter.

func (*IndexMap[K, V]) AllMut 

func (m *IndexMap[K, V]) AllMut() iter.Seq2[K, *V]

AllMut returns each key and a pointer to its value, in order.

AllMut is IndexMap::iter_mut.

func (*IndexMap[K, V]) Clone 

func (m *IndexMap[K, V]) Clone() IndexMap[K, V]

Clone returns a copy of the map. The values are copied as Go copies them.

Clone is the Clone of IndexMap.

func (*IndexMap[K, V]) ContainsKey 

func (m *IndexMap[K, V]) ContainsKey(key K) bool

ContainsKey reports whether the map holds a key.

ContainsKey is IndexMap::contains_key.

func (*IndexMap[K, V]) Drain added in v0.4.0 

func (m *IndexMap[K, V]) Drain() iter.Seq2[K, V]

Drain returns each key and value, in order, and removes them all from the map. The map keeps its capacity. The entries are removed when the loop ends, even when it ends early.

Drain is IndexMap::drain(..).

func (*IndexMap[K, V]) Extend 

func (m *IndexMap[K, V]) Extend(seq iter.Seq2[K, V])

Extend inserts each key and value of seq, in order, as Insert does.

Extend is the Extend of IndexMap.

func (*IndexMap[K, V]) Get 

func (m *IndexMap[K, V]) Get(key K) (V, bool)

Get returns the value of a key, and false when the map does not hold the key.

Get is IndexMap::get.

func (*IndexMap[K, V]) GetIndex 

func (m *IndexMap[K, V]) GetIndex(i int) (K, V, bool)

GetIndex returns the key and the value at an index, and false when the index is not before Len.

GetIndex is IndexMap::get_index.

func (*IndexMap[K, V]) GetIndexMut 

func (m *IndexMap[K, V]) GetIndexMut(i int) (K, *V, bool)

GetIndexMut returns the key and a pointer to the value at an index, and false when the index is not before Len.

GetIndexMut is IndexMap::get_index_mut.

func (*IndexMap[K, V]) GetIndexOf 

func (m *IndexMap[K, V]) GetIndexOf(key K) (int, bool)

GetIndexOf returns the index of a key, and false when the map does not hold the key.

GetIndexOf is IndexMap::get_index_of.

func (*IndexMap[K, V]) GetMut 

func (m *IndexMap[K, V]) GetMut(key K) *V

GetMut returns a pointer to the value of a key, and nil when the map does not hold the key.

GetMut is IndexMap::get_mut.

func (*IndexMap[K, V]) GetOrInsert 

func (m *IndexMap[K, V]) GetOrInsert(key K, value V) *V

GetOrInsert returns a pointer to the value of a key. When the map does not hold the key, GetOrInsert adds it at the end with value.

GetOrInsert is IndexMap::entry(key).or_insert(value), and or_default with the zero value.

func (*IndexMap[K, V]) GetOrInsertFunc 

func (m *IndexMap[K, V]) GetOrInsertFunc(key K, f func() V) *V

GetOrInsertFunc returns a pointer to the value of a key. When the map does not hold the key, GetOrInsertFunc adds it at the end with the value that f returns.

GetOrInsertFunc is IndexMap::entry(key).or_insert_with(f).

func (*IndexMap[K, V]) Insert 

func (m *IndexMap[K, V]) Insert(key K, value V) (V, bool)

Insert sets the value of a key. A new key goes at the end. A key that the map holds keeps its place, and Insert returns its old value and true.

Insert is IndexMap::insert.

func (*IndexMap[K, V]) IsEmpty 

func (m *IndexMap[K, V]) IsEmpty() bool

IsEmpty reports whether the map has no entries.

IsEmpty is IndexMap::is_empty.

func (*IndexMap[K, V]) Keys 

func (m *IndexMap[K, V]) Keys() iter.Seq[K]

Keys returns each key, in order.

Keys is IndexMap::keys.

func (*IndexMap[K, V]) Len 

func (m *IndexMap[K, V]) Len() int

Len returns the number of entries.

Len is IndexMap::len.

func (*IndexMap[K, V]) ShrinkToFit added in v0.4.0 

func (m *IndexMap[K, V]) ShrinkToFit()

ShrinkToFit gives back the capacity that the map grew into, and keeps its entries and their order. A pointer that a method returned into the map is not valid after ShrinkToFit.

ShrinkToFit is IndexMap::shrink_to_fit. A Go map does not shrink, so ShrinkToFit makes the map of indices again, with room for the entries only.

func (*IndexMap[K, V]) Values 

func (m *IndexMap[K, V]) Values() iter.Seq[V]

Values returns each value, in order.

Values is IndexMap::values.

func (*IndexMap[K, V]) ValuesMut 

func (m *IndexMap[K, V]) ValuesMut() iter.Seq[*V]

ValuesMut returns a pointer to each value, in order.

ValuesMut is IndexMap::values_mut.

type IndirectRecursionError 

type IndirectRecursionError struct {
	// Symbols is the cycle, with its first symbol again at the end.
	Symbols []string
}

IndirectRecursionError is the error of a grammar with a rule that derives itself through a chain of rules of one symbol, such as A -> B -> A.

IndirectRecursionError is IndirectRecursionError.

func (*IndirectRecursionError) Error 

func (e *IndirectRecursionError) Error() string

Error returns the text of the error.

type InlinedProductionMap 

type InlinedProductionMap struct {
	Map map[[2]uint32][]uint32
}

InlinedProductionMap maps a production and a step to the productions that inline the symbol of the step.

InlinedProductionMap is InlinedProductionMap. Upstream keys an FxHashMap with the pair, and only looks keys up.

func (*InlinedProductionMap) InlinedProdIDs 

func (m *InlinedProductionMap) InlinedProdIDs(prodID, stepIndex uint32) ([]uint32, bool)

InlinedProdIDs returns the productions that inline the symbol at a step.

InlinedProdIDs is InlinedProductionMap::inlined_prod_ids.

type InputGrammar 

type InputGrammar struct {
	Pool                *RulePool
	Name                StrID
	Variables           []Variable
	ExternalRoots       []RuleID
	ExtraRoots          []RuleID
	ReservedSets        []ReservedWordContext
	SupertypeNames      []StrID
	ConflictNames       [][]StrID
	InlineNames         []StrID
	WordName            StrID // zero when the grammar has no word
	PrecedenceOrderings [][]PrecedenceEntry
}

InputGrammar is the grammar as grammar.json gives it, after the rules that nothing uses are dropped.

InputGrammar is InputGrammar.

func ParseGrammar 

func ParseGrammar(input []byte, diagnostics *[]Diagnostic) (*InputGrammar, error)

ParseGrammar reads grammar.json into an InputGrammar, and appends the warnings that it finds to diagnostics.

ParseGrammar is parse_grammar.

type InternSymbolsError 

type InternSymbolsError struct {
	Kind InternSymbolsErrorKind
	// Name is the symbol that is not defined, for every kind but
	// InternSymbolsHiddenStartRule.
	Name string
}

InternSymbolsError is an error of the pass that interns the symbols. Its text is the text of upstream.

InternSymbolsError is InternSymbolsError.

func (*InternSymbolsError) Error 

func (e *InternSymbolsError) Error() string

Error returns the text of the error.

type InternSymbolsErrorKind 

type InternSymbolsErrorKind uint8

InternSymbolsErrorKind is the kind of an error of the pass that interns the symbols. 

const (
	InternSymbolsHiddenStartRule InternSymbolsErrorKind = iota
	InternSymbolsUndefined
	InternSymbolsUndefinedSupertype
	InternSymbolsUndefinedConflict
	InternSymbolsUndefinedWordToken
)

The kinds of error, in the order of upstream.

type Interpretation 

type Interpretation struct {
	PrecedingSymbols      []string
	VariableName          string
	ProductionStepSymbols []string
	StepIndex             uint32
	Done                  bool
	ConflictingLookahead  string
	Precedence            string
	HasPrecedence         bool
	Associativity         string
	HasAssociativity      bool
	RequiresEOFLookahead  bool
}

Interpretation is one way in which the parser can read the input at a conflict. HasPrecedence and HasAssociativity are false for the None of upstream.

Interpretation is Interpretation. Upstream also derives Error for it. The port gives it only its text.

func (*Interpretation) String 

func (i *Interpretation) String() string

String returns the text of the interpretation.

String is the Display of Interpretation.

type InvalidSupertypeError 

type InvalidSupertypeError struct {
	Supertype string
	// Child is the name of a hidden child that can expand into several
	// nodes, or empty when there is none. A variable never has an empty
	// name, so empty stands for the None of upstream.
	Child string
}

InvalidSupertypeError is the error of a supertype that can have more than one visible child.

Upstream wraps it in VariableInfoError::InvalidSupertype, which is transparent, so GetVariableInfo returns it as it is.

InvalidSupertypeError is InvalidSupertypeError.

func (*InvalidSupertypeError) Error 

func (e *InvalidSupertypeError) Error() string

Error returns the text of the error.

Error is the Display of InvalidSupertypeError.

type ItemKeyMap 

type ItemKeyMap struct {
	// contains filtered or unexported fields
}

ItemKeyMap holds the identity keys of every production and every position of the dot in a grammar: all the productions of the grammar, every inlined production, and the start production.

ItemKeyMap is ItemKeyMap.

func NewItemKeyMap 

func NewItemKeyMap(grammar *SyntaxGrammar, strPool *StrPool) *ItemKeyMap

NewItemKeyMap returns the identity keys of the productions of a grammar.

NewItemKeyMap is ItemKeyMap::new.

func (*ItemKeyMap) KeysFor 

func (m *ItemKeyMap) KeysFor(id uint32) []DotKeys

KeysFor returns the keys of a production of the grammar, by the position of the dot.

KeysFor is ItemKeyMap::keys_for.

func (*ItemKeyMap) StartKeys 

func (m *ItemKeyMap) StartKeys() []DotKeys

StartKeys returns the keys of the start production.

StartKeys is ItemKeyMap::start_keys.

type LargeCharacterSet 

type LargeCharacterSet struct {
	Symbol    Symbol
	HasSymbol bool
	Chars     CharacterSet
}

LargeCharacterSet is a large character set, and the token that it belongs to. HasSymbol is false for the None of upstream, which is a set that is not in the main token.

LargeCharacterSet is (Option<Symbol>, CharacterSet).

type LexAdvance 

type LexAdvance struct {
	Chars  CharacterSet
	Action AdvanceAction
}

LexAdvance is a set of characters and the action of the lexer on them.

LexAdvance is (CharacterSet, AdvanceAction).

type LexState 

type LexState struct {
	AcceptAction    Symbol
	HasAcceptAction bool
	EOFAction       AdvanceAction
	HasEOFAction    bool
	AdvanceActions  []LexAdvance
}

LexState is a state of a lex table. HasAcceptAction and HasEOFAction are false for the None of upstream.

LexState is LexState. The zero LexState is LexState::default.

func (*LexState) Equal 

func (s *LexState) Equal(other *LexState) bool

Equal reports whether two states are the same.

Equal is the PartialEq of LexState.

type LexStateID 

type LexStateID = uint32

LexStateID is the index of a state in a lex table.

LexStateID is LexStateId.

type LexTable 

type LexTable struct {
	States []LexState
}

LexTable is a lex table.

LexTable is LexTable. The zero LexTable is LexTable::default.

type LexTables 

type LexTables struct {
	MainLexTable       LexTable
	KeywordLexTable    LexTable
	LargeCharacterSets []LargeCharacterSet
}

LexTables holds the lex tables of a grammar and its large character sets.

LexTables is LexTables.

func BuildLexTable 

func BuildLexTable(
	parseTable *ParseTable,
	syntaxGrammar *SyntaxGrammar,
	lexicalGrammar *LexicalGrammar,
	keywords *TokenSet,
	coincidentTokenIndex *CoincidentTokenIndex,
	tokenConflictMap *TokenConflictMap,
) LexTables

BuildLexTable builds the main lex table and the keyword lex table, and sets the lex state of each state of the parse table.

BuildLexTable is build_lex_table.

type LexicalGrammar 

type LexicalGrammar struct {
	Nfa       Nfa
	Variables []LexicalVariable
}

LexicalGrammar is the grammar of the lexer: one NFA for every token, and the tokens.

LexicalGrammar is LexicalGrammar.

func (*LexicalGrammar) VariableIndexForNfaState 

func (g *LexicalGrammar) VariableIndexForNfaState(stateID uint32) int

VariableIndexForNfaState returns the index of the variable that a state of the NFA belongs to.

VariableIndexForNfaState is LexicalGrammar::variable_index_for_nfa_state.

func (*LexicalGrammar) VariableIndicesForNfaStates 

func (g *LexicalGrammar) VariableIndicesForNfaStates(stateIDs []uint32) iter.Seq[int]

VariableIndicesForNfaStates returns the index of the variable of each state. It skips an index that is the same as the one before it.

VariableIndicesForNfaStates is LexicalGrammar::variable_indices_for_nfa_states.

type LexicalToken 

type LexicalToken struct {
	// Name is a made name for an anonymous token, or the name of the rule of
	// a variable that became a token.
	Name StrID
	Kind VariableType
	// Root is the rule in the pool that defines the token.
	Root RuleID
}

LexicalToken is a token that is taken out of the input grammar and is not yet expanded into the NFA of the lexer. Root is still the rule of the token in the pool.

LexicalToken is LexicalToken.

type LexicalVariable 

type LexicalVariable struct {
	Name               StrID
	Kind               VariableType
	ImplicitPrecedence int32
	StartState         uint32
}

LexicalVariable is a token of the lexical grammar.

LexicalVariable is LexicalVariable.

type LookaheadSetID 

type LookaheadSetID uint32

LookaheadSetID is the id of a lookahead set in a LookaheadSetPool. The ids are canonical, so two ids are equal exactly when their sets are equal. The zero LookaheadSetID is the empty set.

LookaheadSetID is LookaheadSetId, and its Default is LookaheadSetPool::EMPTY. 

const LookaheadSetPoolEmpty LookaheadSetID = 0

LookaheadSetPoolEmpty is the id of the empty set.

LookaheadSetPoolEmpty is LookaheadSetPool::EMPTY.

type LookaheadSetPool 

type LookaheadSetPool struct {
	// contains filtered or unexported fields
}

LookaheadSetPool interns lookahead sets. The entries of an item set hold ids.

LookaheadSetPool is LookaheadSetPool. Upstream holds the sets in an IndexSet. Go holds them in a slice, with a map from the Key of each set to its index.

func NewLookaheadSetPool 

func NewLookaheadSetPool() *LookaheadSetPool

NewLookaheadSetPool returns a pool that holds the empty set, with the id LookaheadSetPoolEmpty.

NewLookaheadSetPool is LookaheadSetPool::new.

func (*LookaheadSetPool) Get 

func (p *LookaheadSetPool) Get(id LookaheadSetID) *TokenSet

Get returns the set of an id. Do not change the set.

Get is LookaheadSetPool::get.

func (*LookaheadSetPool) Insert 

func (p *LookaheadSetPool) Insert(id LookaheadSetID, symbol Symbol) LookaheadSetID

Insert returns the id of the set of id with symbol added.

Insert is LookaheadSetPool::insert.

func (*LookaheadSetPool) Intern 

func (p *LookaheadSetPool) Intern(set TokenSet) LookaheadSetID

Intern returns the id of a set, and adds the set to the pool when it is new. The pool takes the set, so do not change it after the call.

Intern is LookaheadSetPool::intern.

func (*LookaheadSetPool) InternRef 

func (p *LookaheadSetPool) InternRef(set *TokenSet) LookaheadSetID

InternRef returns the id of a set, and adds a copy of the set to the pool when it is new.

InternRef is LookaheadSetPool::intern_ref.

func (*LookaheadSetPool) Singleton 

func (p *LookaheadSetPool) Singleton(symbol Symbol) LookaheadSetID

Singleton returns the id of the set that holds only symbol.

Singleton is LookaheadSetPool::singleton.

func (*LookaheadSetPool) Union 

func (p *LookaheadSetPool) Union(left, right LookaheadSetID) LookaheadSetID

Union returns the id of the union of two sets.

Union is LookaheadSetPool::union.

type MetadataParams 

type MetadataParams struct {
	Precedence        Precedence
	Associativity     Associativity
	DynamicPrecedence int32
	Alias             Alias
	HasAlias          bool
	Field             StrID
	IsToken           bool
	IsMainToken       bool
}

MetadataParams is the metadata of a rule. A zero Alias.Value and a zero Field stand for the None of upstream.

MetadataParams is MetadataParams.

type Nfa 

type Nfa struct {
	States []NfaState
}

Nfa is a nondeterministic finite automaton, the tokens of a lexical grammar. The zero Nfa has no states.

Nfa is Nfa, and the zero Nfa is Nfa::new.

func (*Nfa) LastStateID 

func (n *Nfa) LastStateID() uint32

LastStateID returns the id of the last state. It panics when the NFA has no states.

LastStateID is Nfa::last_state_id.

type NfaCursor 

type NfaCursor struct {
	// contains filtered or unexported fields
}

NfaCursor is a set of states of an NFA, which moves on the characters of the input.

NfaCursor is NfaCursor.

func NewNfaCursor 

func NewNfaCursor(nfa *Nfa, states []uint32) *NfaCursor

NewNfaCursor returns a cursor on an NFA at a set of states. It takes states, and the caller must not use it after.

NewNfaCursor is NfaCursor::new.

func (*NfaCursor) AddStates 

func (c *NfaCursor) AddStates(newStateIDs []uint32)

AddStates adds states to the cursor. A split state is not added, but the two states that it splits to are. It takes newStateIDs, and the caller must not use it after.

AddStates is NfaCursor::add_states.

func (*NfaCursor) Completions 

func (c *NfaCursor) Completions() iter.Seq2[int, int32]

Completions returns the variable index and the precedence of each state of the cursor that accepts.

Completions is NfaCursor::completions.

func (*NfaCursor) ForceReset 

func (c *NfaCursor) ForceReset(states []uint32)

ForceReset moves the cursor to exactly a set of states. It takes states, and the caller must not use it after.

ForceReset is NfaCursor::force_reset.

func (*NfaCursor) Reset 

func (c *NfaCursor) Reset(states []uint32)

Reset moves the cursor to a set of states, and to the states that they split to. It takes states, and the caller must not use it after.

Reset is NfaCursor::reset.

func (*NfaCursor) TransitionChars 

func (c *NfaCursor) TransitionChars() iter.Seq2[CharacterSet, bool]

TransitionChars returns the characters of each move from the states of the cursor, and whether the move is a separator. The caller must not change a set.

TransitionChars is NfaCursor::transition_chars.

func (*NfaCursor) Transitions 

func (c *NfaCursor) Transitions() []NfaTransition

Transitions returns the moves from the states of the cursor, grouped so that no two moves share a character.

Transitions is NfaCursor::transitions.

func (*NfaCursor) TransitionsAndAnySep 

func (c *NfaCursor) TransitionsAndAnySep() ([]NfaTransition, bool)

TransitionsAndAnySep returns what Transitions returns, and whether any move is a separator.

TransitionsAndAnySep is NfaCursor::transitions_and_any_sep.

type NfaState 

type NfaState struct {
	Kind          NfaStateKind
	Chars         CharacterSet
	StateID       uint32
	IsSep         bool
	Precedence    int32
	Left          uint32
	Right         uint32
	VariableIndex int
}

NfaState is a state of an NFA.

NfaState is NfaState, an enum with data upstream. The fields that a kind uses are:

  • NfaAdvance: Chars, StateID, IsSep and Precedence.
  • NfaSplit: Left and Right.
  • NfaAccept: VariableIndex and Precedence.

type NfaStateKind 

type NfaStateKind uint8

NfaStateKind is the kind of a state of an NFA. 

const (
	NfaAdvance NfaStateKind = iota
	NfaSplit
	NfaAccept
)

The kinds of NFA state, in the order of the enum NfaState.

type NfaTransition 

type NfaTransition struct {
	Characters  CharacterSet
	IsSeparator bool
	Precedence  int32
	States      []uint32
}

NfaTransition is a move of an NFA cursor on a set of characters.

NfaTransition is NfaTransition.

type NonTerminalIndex 

type NonTerminalIndex uint32

NonTerminalIndex is the index of a non-terminal. It is a type of the macro symbol_index.

func (NonTerminalIndex) Symbol 

func (i NonTerminalIndex) Symbol() Symbol

Symbol returns the symbol of the index.

type NonTerminalWordTokenError 

type NonTerminalWordTokenError struct {
	SymbolName string
	// ConflictingSymbolName is the name of a rule with the same body as the
	// word, or empty when no rule has it.
	ConflictingSymbolName string
}

NonTerminalWordTokenError is the error of a word token that is a non-terminal.

NonTerminalWordTokenError is NonTerminalWordTokenError.

func (*NonTerminalWordTokenError) Error 

func (e *NonTerminalWordTokenError) Error() string

Error returns the text of the error. Upstream ends it with a newline.

type NonterminalEntries added in v0.4.0 

type NonterminalEntries = ParseStateEntries[GotoAction]

NonterminalEntries holds the non-terminal entries of a parse state: what to do after a reduction to each non-terminal.

NonterminalEntries is NonterminalEntries.

type OptLevel 

type OptLevel uint32

OptLevel is a set of flags for the optimizations of the generator.

OptLevel is OptLevel, a bitflags type upstream. Its Default is OptLevelMergeStates. 

const (
	// OptLevelMergeStates merges the parse states that are compatible.
	OptLevelMergeStates OptLevel = 1 << 0
)

The optimizations.

func (OptLevel) Contains 

func (o OptLevel) Contains(other OptLevel) bool

Contains reports whether o holds every flag of other.

Contains is OptLevel::contains.

type ParamsID 

type ParamsID uint32

ParamsID is the index of metadata in a rule pool.

ParamsID is ParamsId.

type ParseAction 

type ParseAction struct {
	Kind              ParseActionKind
	State             ParseStateID
	IsRepetition      bool
	Symbol            Symbol
	ChildCount        uint16
	DynamicPrecedence int32
	ProductionID      uint16
}

ParseAction is an action of the parser for a lookahead token.

ParseAction is ParseAction, an enum with data upstream. The fields that a kind uses are:

  • ParseActionShift: State and IsRepetition.
  • ParseActionReduce: Symbol, ChildCount, DynamicPrecedence and ProductionID.

A field that the kind does not use is zero, so two actions compare with ==.

type ParseActionKind 

type ParseActionKind uint8

ParseActionKind is the kind of a parse action. 

const (
	ParseActionAccept ParseActionKind = iota
	ParseActionShift
	ParseActionShiftExtra
	ParseActionRecover
	ParseActionReduce
)

The kinds of parse action, in the order of the enum ParseAction.

type ParseGrammarError 

type ParseGrammarError struct {
	Kind ParseGrammarErrorKind
	// Detail is the text of a JSON error, or the name of the rule of
	// ParseGrammarUnexpectedRule.
	Detail string
}

ParseGrammarError is an error of ParseGrammar. Its text is the text of upstream, except for a JSON error, whose text comes from encoding/json.

ParseGrammarError is ParseGrammarError.

func (*ParseGrammarError) Error 

func (e *ParseGrammarError) Error() string

Error returns the text of the error.

type ParseGrammarErrorKind 

type ParseGrammarErrorKind uint8

ParseGrammarErrorKind is the kind of an error of ParseGrammar. 

const (
	ParseGrammarSerialization ParseGrammarErrorKind = iota
	ParseGrammarInvalidExtra
	ParseGrammarUnexpected
	ParseGrammarInvalidReservedWordSet
	ParseGrammarUnexpectedRule
)

The kinds of error, in the order of upstream.

type ParseItem 

type ParseItem struct {
	// VariableIndex is the index of the parent rule within the grammar.
	VariableIndex uint32
	// StepIndex is the number of symbols that have already been matched.
	StepIndex uint32
	// ProdID is the id of the production being matched.
	ProdID uint32
	// Keys is the identity keys of the production, indexed by StepIndex.
	Keys []DotKeys
	// HasPrecedingInheritedFields is true when any of the already-matched
	// children were hidden nodes and had fields. Ordinarily, a parse item's
	// behavior is not affected by the symbols of its preceding children; it
	// only needs to keep track of their fields and aliases.
	//
	// Take for example these two items:
	//   X -> a b • c
	//   X -> a g • c
	//
	// They can be considered equivalent, for the purposes of parse table
	// generation, because they entail the same actions. But if this flag is
	// true, then the item's set of inherited fields may depend on the specific
	// symbols of its preceding children.
	HasPrecedingInheritedFields bool
}

ParseItem is a match of one production of a grammar that is in progress.

ParseItem is ParseItem.

func StartParseItem 

func StartParseItem(keyMap *ItemKeyMap) ParseItem

StartParseItem returns the item of the start production, with the dot at the start.

StartParseItem is ParseItem::start.

func (*ParseItem) Associativity 

func (i *ParseItem) Associativity(grammar *SyntaxGrammar) Associativity

Associativity returns the associativity of the step before the dot, and AssociativityNone when the dot is at the start.

Associativity is ParseItem::associativity.

func (ParseItem) Compare 

func (i ParseItem) Compare(other ParseItem) int

Compare orders two items by step index, variable index, the rank of their content, HasPrecedingInheritedFields, and then, when that is true, the class of their preceding symbols.

Compare is the Ord of ParseItem.

func (*ParseItem) Equal 

func (i *ParseItem) Equal(other *ParseItem) bool

Equal reports whether two items are equal.

Equal is the PartialEq of ParseItem.

func (*ParseItem) IsAugmented 

func (i *ParseItem) IsAugmented() bool

IsAugmented reports whether the item matches the start production.

IsAugmented is ParseItem::is_augmented.

func (*ParseItem) IsDone 

func (i *ParseItem) IsDone() bool

IsDone reports whether the dot is at the end of the production.

IsDone is ParseItem::is_done.

func (*ParseItem) Key 

func (i *ParseItem) Key() ParseItemKey

Key returns the key of the item.

The already-matched children don't play any role in the parse state for this item, unless any of the following are true:

  • the children have fields
  • the children have aliases
  • the children are hidden and represent rules that have fields.

See the docs for HasPrecedingInheritedFields. Preceding symbols participate only in EqWithSyms.

Key stands in for the Hash of ParseItem.

func (*ParseItem) Precedence 

func (i *ParseItem) Precedence(grammar *SyntaxGrammar) Precedence

Precedence returns the precedence of the step before the dot, and no precedence when the dot is at the start.

Precedence is ParseItem::precedence.

func (*ParseItem) PrevStep 

func (i *ParseItem) PrevStep(grammar *SyntaxGrammar) (ProductionStep, bool)

PrevStep returns the step before the dot, and false when the dot is at the start.

PrevStep is ParseItem::prev_step.

func (*ParseItem) Production 

func (i *ParseItem) Production(grammar *SyntaxGrammar) ProdRef

Production returns the production being matched.

Production is ParseItem::production.

func (*ParseItem) Step 

func (i *ParseItem) Step(grammar *SyntaxGrammar) (ProductionStep, bool)

Step returns the step after the dot, and false when the dot is at the end.

Step is ParseItem::step.

func (*ParseItem) SubstituteProduction 

func (i *ParseItem) SubstituteProduction(prodID uint32, keys []DotKeys) ParseItem

SubstituteProduction returns an item identical to this one, but with a different production. This is used when dynamically "inlining" certain symbols in a production.

SubstituteProduction is ParseItem::substitute_production.

func (*ParseItem) Successor 

func (i *ParseItem) Successor() ParseItem

Successor returns an item like this one, but advanced by one step.

Successor is ParseItem::successor.

func (*ParseItem) Symbol 

func (i *ParseItem) Symbol(grammar *SyntaxGrammar) (Symbol, bool)

Symbol returns the symbol after the dot, and false when the dot is at the end.

Symbol is ParseItem::symbol.

type ParseItemDisplay 

type ParseItemDisplay struct {
	Item    *ParseItem
	Syntax  *SyntaxGrammar
	Lexical *LexicalGrammar
	Pool    *StrPool
}

ParseItemDisplay shows an item as text, for the messages of the generator.

ParseItemDisplay is ParseItemDisplay.

func (ParseItemDisplay) String 

func (d ParseItemDisplay) String() string

String returns the text of the item.

String is the Display of ParseItemDisplay.

type ParseItemKey 

type ParseItemKey struct {
	VariableIndex               uint32
	StepIndex                   uint32
	HasPrecedingInheritedFields bool
	Class                       uint32
}

ParseItemKey is a value that is the same for two items exactly when they are equal, so it can be the key of a Go map.

ParseItemKey is what the Hash of ParseItem hashes.

type ParseItemSet 

type ParseItemSet struct {
	Entries []ParseItemSetEntry
}

ParseItemSet is a set of matches of productions of a grammar that are in progress, sorted by item.

For each in-progress match, a set of "lookaheads" (tokens that are allowed to *follow* the in-progress rule) are included. This object corresponds directly to a state in the final parse table.

ParseItemSet is ParseItemSet. The zero ParseItemSet is ParseItemSet::default.

func (*ParseItemSet) Core 

func (s *ParseItemSet) Core() ParseItemSetCore

Core returns the items of the set, without their lookaheads.

Core is ParseItemSet::core.

func (*ParseItemSet) Insert 

func (s *ParseItemSet) Insert(item ParseItem) *ParseItemSetEntry

Insert adds an item to the set in its sorted place, with an empty lookahead set, when the set does not hold it yet. It returns the entry of the item. The pointer is valid until the next Insert.

Insert is ParseItemSet::insert.

func (*ParseItemSet) Key 

func (s *ParseItemSet) Key() string

Key returns a string that is the same for two item sets exactly when they are equal, for a map key. Two sets are equal when their entries are equal in order: the items, the lookahead ids and the reserved word sets.

Key stands in for the Hash and the Eq of ParseItemSet.

type ParseItemSetBuilder 

type ParseItemSetBuilder struct {
	KeyMap     *ItemKeyMap
	Lookaheads *LookaheadSetPool
	// contains filtered or unexported fields
}

ParseItemSetBuilder computes the transitive closure of item sets.

ParseItemSetBuilder is ParseItemSetBuilder.

func NewParseItemSetBuilder 

func NewParseItemSetBuilder(syntaxGrammar *SyntaxGrammar, lexicalGrammar *LexicalGrammar, inlines *InlinedProductionMap, keyMap *ItemKeyMap) *ParseItemSetBuilder

NewParseItemSetBuilder returns a builder for a grammar, with the FIRST and LAST sets of its symbols and the additions of each non-terminal.

NewParseItemSetBuilder is ParseItemSetBuilder::new.

func (*ParseItemSetBuilder) FirstSet 

func (b *ParseItemSetBuilder) FirstSet(symbol Symbol) *TokenSet

FirstSet returns the FIRST set of a symbol. It panics when the builder has no FIRST set for the symbol, as upstream does.

FirstSet is ParseItemSetBuilder::first_set.

func (*ParseItemSetBuilder) LastSet 

func (b *ParseItemSetBuilder) LastSet(symbol Symbol) *TokenSet

LastSet returns the LAST set of a symbol. It panics when the builder has no LAST set for the symbol, as upstream does.

LastSet is ParseItemSetBuilder::last_set.

func (*ParseItemSetBuilder) ReservedFirstSet 

func (b *ParseItemSetBuilder) ReservedFirstSet(symbol Symbol) (*TokenSet, bool)

ReservedFirstSet returns the reserved words that can start a symbol, and false when the builder has none for the symbol.

ReservedFirstSet is ParseItemSetBuilder::reserved_first_set.

func (*ParseItemSetBuilder) TransitiveClosure 

func (b *ParseItemSetBuilder) TransitiveClosure(itemSet *ParseItemSet) ParseItemSet

TransitiveClosure returns the closure of an item set: its items, with each symbol that is inlined replaced by its productions, and the items that each non-terminal after a dot adds.

TransitiveClosure is ParseItemSetBuilder::transitive_closure.

type ParseItemSetCore 

type ParseItemSetCore struct {
	Entries []ParseItem
}

ParseItemSetCore is like a ParseItemSet, but without the lookahead information. Parse states with the same core are candidates for merging.

ParseItemSetCore is ParseItemSetCore.

func (*ParseItemSetCore) Key 

func (c *ParseItemSetCore) Key() string

Key returns a string that is the same for two cores exactly when they are equal, for a map key.

Key stands in for the Hash and the Eq of ParseItemSetCore.

type ParseItemSetDisplay 

type ParseItemSetDisplay struct {
	Set        *ParseItemSet
	Syntax     *SyntaxGrammar
	Lexical    *LexicalGrammar
	Pool       *StrPool
	Lookaheads *LookaheadSetPool
}

ParseItemSetDisplay shows an item set as text, for the messages of the generator.

ParseItemSetDisplay is ParseItemSetDisplay.

func (ParseItemSetDisplay) String 

func (d ParseItemSetDisplay) String() string

String returns the text of the item set, one entry on each line.

String is the Display of ParseItemSetDisplay.

type ParseItemSetEntry 

type ParseItemSetEntry struct {
	Item                     ParseItem
	Lookaheads               LookaheadSetID
	FollowingReservedWordSet ReservedWordSetID
}

ParseItemSetEntry is an item of an item set, with its lookahead set and the reserved word set that can follow it.

ParseItemSetEntry is ParseItemSetEntry.

type ParseState 

type ParseState struct {
	ID                 ParseStateID
	TerminalEntries    TerminalEntries
	NonterminalEntries NonterminalEntries
	ReservedWords      TokenSet
	LexStateID         LexStateID
	ExternalLexStateID LexStateID
	CoreID             uint32
	HasEOFGatedReduce  bool
}

ParseState is a state of the parse table.

ParseState is ParseState. The zero ParseState is ParseState::default. A ParseState holds IndexMaps, so it must not be copied while it is in use (see IndexMap).

func (*ParseState) IsEndOfNonTerminalExtra 

func (s *ParseState) IsEndOfNonTerminalExtra() bool

IsEndOfNonTerminalExtra reports whether the state ends a non-terminal extra.

IsEndOfNonTerminalExtra is ParseState::is_end_of_non_terminal_extra.

func (*ParseState) ReferencedStates added in v0.4.0 

func (s *ParseState) ReferencedStates(pool *ActionListPool) iter.Seq[ParseStateID]

ReferencedStates returns each state that s refers to: the target of each shift action of the terminal entries, in the order of the entries, then the target of each goto action of the non-terminal entries.

ReferencedStates is ParseState::referenced_states.

func (*ParseState) UpdateNonterminalReferences 

func (s *ParseState) UpdateNonterminalReferences(f func(ParseStateID, *ParseState) ParseStateID)

UpdateNonterminalReferences changes the target of each goto action of the non-terminal entries to what f returns for it. f gets the target and the state.

UpdateNonterminalReferences is ParseState::update_nonterminal_references.

type ParseStateEntries added in v0.4.0 

type ParseStateEntries[V any] struct {
	// contains filtered or unexported fields
}

ParseStateEntries holds the entries of a parse state, one for each symbol, in the order in which they were added. The zero ParseStateEntries is empty and ready to use.

ParseStateEntries is ParseStateEntries, a Vec<(Symbol, V)>.

func (*ParseStateEntries[V]) All added in v0.4.0 

func (e *ParseStateEntries[V]) All() iter.Seq2[Symbol, V]

All returns each symbol and its value, in order.

All is ParseStateEntries::iter, and the IntoIterator of ParseStateEntries.

func (*ParseStateEntries[V]) AllMut added in v0.4.0 

func (e *ParseStateEntries[V]) AllMut() iter.Seq2[Symbol, *V]

AllMut returns each symbol and a pointer to its value, in order.

AllMut is ParseStateEntries::iter_mut.

func (*ParseStateEntries[V]) ContainsKey added in v0.4.0 

func (e *ParseStateEntries[V]) ContainsKey(symbol Symbol) bool

ContainsKey reports whether there is an entry for symbol.

ContainsKey is ParseStateEntries::contains_key.

func (*ParseStateEntries[V]) Get added in v0.4.0 

func (e *ParseStateEntries[V]) Get(symbol Symbol) (V, bool)

Get returns the value of symbol, and false when there is no entry for symbol.

Get is ParseStateEntries::get.

func (*ParseStateEntries[V]) GetIndex added in v0.4.0 

func (e *ParseStateEntries[V]) GetIndex(i int) (Symbol, V, bool)

GetIndex returns the symbol and the value at an index, and false when the index is not before Len.

GetIndex is ParseStateEntries::get_index.

func (*ParseStateEntries[V]) GetIndexMut added in v0.4.0 

func (e *ParseStateEntries[V]) GetIndexMut(i int) (Symbol, *V, bool)

GetIndexMut returns the symbol and a pointer to the value at an index, and false when the index is not before Len.

GetIndexMut is ParseStateEntries::get_index_mut.

func (*ParseStateEntries[V]) Insert added in v0.4.0 

func (e *ParseStateEntries[V]) Insert(symbol Symbol, value V)

Insert sets the value for symbol. An entry that exists keeps its place.

Insert is ParseStateEntries::insert.

func (*ParseStateEntries[V]) InsertIfMissing added in v0.4.0 

func (e *ParseStateEntries[V]) InsertIfMissing(symbol Symbol, value V)

InsertIfMissing adds an entry for symbol, unless there is one.

InsertIfMissing is ParseStateEntries::insert_if_missing.

func (*ParseStateEntries[V]) Keys added in v0.4.0 

func (e *ParseStateEntries[V]) Keys() iter.Seq[Symbol]

Keys returns each symbol, in order.

Keys is ParseStateEntries::keys.

func (*ParseStateEntries[V]) Len added in v0.4.0 

func (e *ParseStateEntries[V]) Len() int

Len returns the number of entries.

Len is ParseStateEntries::len.

func (*ParseStateEntries[V]) Push added in v0.4.0 

func (e *ParseStateEntries[V]) Push(symbol Symbol, value V)

Push adds an entry for symbol, which must not have one yet.

Push is ParseStateEntries::push. Upstream checks the symbol only with debug_assert.

func (*ParseStateEntries[V]) ReserveExact added in v0.4.0 

func (e *ParseStateEntries[V]) ReserveExact(additional int)

ReserveExact makes room for additional more entries, with no spare capacity.

ReserveExact is ParseStateEntries::reserve_exact.

func (*ParseStateEntries[V]) ShrinkToFit added in v0.4.0 

func (e *ParseStateEntries[V]) ShrinkToFit()

ShrinkToFit gives back the capacity that the entries grew into.

ShrinkToFit is ParseStateEntries::shrink_to_fit.

func (*ParseStateEntries[V]) Values added in v0.4.0 

func (e *ParseStateEntries[V]) Values() iter.Seq[V]

Values returns each value, in order.

Values is ParseStateEntries::values.

func (*ParseStateEntries[V]) ValuesMut added in v0.4.0 

func (e *ParseStateEntries[V]) ValuesMut() iter.Seq[*V]

ValuesMut returns a pointer to each value, in order.

ValuesMut is ParseStateEntries::values_mut.

type ParseStateID 

type ParseStateID = uint32

ParseStateID is the index of a state in a parse table.

ParseStateID is ParseStateId.

type ParseStateInfo 

type ParseStateInfo struct {
	// PrecedingSymbolsByID holds, for each state, an example sequence of
	// symbols that leads to the state. The report of a conflict uses it.
	PrecedingSymbolsByID [][]Symbol
	// Lookaheads is the pool of the lookahead sets of the item sets.
	Lookaheads *LookaheadSetPool
	// contains filtered or unexported fields
}

ParseStateInfo is what the builder of the parse table knows about each state, for the report of the states of a symbol.

ParseStateInfo is ParseStateInfo. Upstream keeps the item sets in an IndexMap from the item set to its state id, and reads it only by index. Each state id is the index of its item set, so itemSetsByIDs is the keys of that map in order.

func (*ParseStateInfo) ItemSet 

func (i *ParseStateInfo) ItemSet(id ParseStateID) *ParseItemSet

ItemSet returns the item set of a state, before the closure.

ItemSet is ParseStateInfo::item_set.

type ParseTable 

type ParseTable struct {
	States                     []ParseState
	ActionLists                ActionListPool
	Symbols                    []Symbol
	ProductionInfos            []ProductionInfo
	MaxAliasedProductionLength int
	ExternalLexStates          []TokenSet
}

ParseTable is the parse table.

ParseTable is ParseTable. The zero ParseTable is ParseTable::default.

func (*ParseTable) RemapTerminalReferences added in v0.4.0 

func (t *ParseTable) RemapTerminalReferences(f func(ParseStateID) ParseStateID)

RemapTerminalReferences changes the target of each shift action to what f returns for it. It changes only the lists of the pool that a state uses.

RemapTerminalReferences is ParseTable::remap_terminal_references.

type ParseTableBuilderError 

type ParseTableBuilderError struct {
	Kind           ParseTableBuilderErrorKind
	Conflict       *ConflictError
	AmbiguousExtra *AmbiguousExtraError
	Name           string
	StateCount     int
}

ParseTableBuilderError is an error of the builder of the parse table. Its text is the text of upstream.

ParseTableBuilderError is ParseTableBuilderError, an enum with data upstream. The fields that a kind uses are:

  • ParseTableBuilderConflict: Conflict.
  • ParseTableBuilderAmbiguousExtra: AmbiguousExtra.
  • ParseTableBuilderImproperNonTerminalExtra: Name.
  • ParseTableBuilderStateCount: StateCount.

Upstream also derives Serialize and Deserialize for the error types of this file. The port does not.

func (*ParseTableBuilderError) Error 

func (e *ParseTableBuilderError) Error() string

Error returns the text of the error.

func (*ParseTableBuilderError) Unwrap 

func (e *ParseTableBuilderError) Unwrap() error

Unwrap returns the error in the error: the *ConflictError or the *AmbiguousExtraError, and nil for the other kinds.

type ParseTableBuilderErrorKind 

type ParseTableBuilderErrorKind uint8

ParseTableBuilderErrorKind is the kind of an error of the builder of the parse table. 

const (
	ParseTableBuilderConflict ParseTableBuilderErrorKind = iota
	ParseTableBuilderAmbiguousExtra
	ParseTableBuilderImproperNonTerminalExtra
	ParseTableBuilderStateCount
)

The kinds of error, in the order of upstream.

type ParseTableEntry 

type ParseTableEntry struct {
	Actions  ActionList
	Reusable bool
}

ParseTableEntry is the entry of the parse table for a state and a terminal, while the table is built.

ParseTableEntry is ParseTableEntry.

func NewParseTableEntry 

func NewParseTableEntry() ParseTableEntry

NewParseTableEntry returns an entry with no actions that is reusable.

NewParseTableEntry is ParseTableEntry::new.

func (ParseTableEntry) Clone 

func (e ParseTableEntry) Clone() ParseTableEntry

Clone returns a copy of the entry.

Clone is the Clone of ParseTableEntry.

type PatternSpan 

type PatternSpan struct {
	Start uint32
	End   uint32
}

PatternSpan is a range of bytes in a pattern.

PatternSpan is PatternSpan.

type Precedence 

type Precedence struct {
	Kind    PrecedenceKind
	Integer int32
	Name    StrID
}

Precedence is a precedence: none, a number, or a name.

Precedence is Precedence, an enum with data upstream.

type PrecedenceEntry 

type PrecedenceEntry struct {
	Kind  PrecedenceEntryKind
	Value StrID
}

PrecedenceEntry is an entry of a list in precedences: the name of a precedence, or the name of a symbol.

PrecedenceEntry is PrecedenceEntry, an enum with data upstream.

type PrecedenceEntryKind 

type PrecedenceEntryKind uint8

PrecedenceEntryKind is the kind of an entry of a precedence list. 

const (
	PrecedenceEntryName PrecedenceEntryKind = iota
	PrecedenceEntrySymbol
)

The kinds of entry, in the order of upstream.

type PrecedenceKind 

type PrecedenceKind uint8

PrecedenceKind is the kind of a precedence. 

const (
	PrecedenceNone PrecedenceKind = iota
	PrecedenceInteger
	PrecedenceName
)

The kinds of precedence, in the order of upstream.

type PreparedGrammar 

type PreparedGrammar struct {
	SyntaxGrammar  SyntaxGrammar
	LexicalGrammar LexicalGrammar
	Inlines        InlinedProductionMap
	DefaultAliases AliasMap
	StrPool        *StrPool
}

PreparedGrammar is the grammar split into the parts that the builder of the tables reads.

PreparedGrammar is PreparedGrammar.

func PrepareGrammar 

func PrepareGrammar(g *InputGrammar, diagnostics *[]Diagnostic) (*PreparedGrammar, error)

PrepareGrammar splits a grammar into the parts that the builder of the tables reads. It runs each pass in order, and it changes g.

PrepareGrammar is prepare_grammar.

type ProcessInlinesError 

type ProcessInlinesError struct {
	Kind ProcessInlinesErrorKind
	Name string
}

ProcessInlinesError is an error of the pass that processes the inlined variables. Its text is the text of upstream.

ProcessInlinesError is ProcessInlinesError.

func (*ProcessInlinesError) Error 

func (e *ProcessInlinesError) Error() string

Error returns the text of the error.

type ProcessInlinesErrorKind 

type ProcessInlinesErrorKind uint8

ProcessInlinesErrorKind is the kind of an error of the pass that processes the inlined variables. 

const (
	ProcessInlinesExternalToken ProcessInlinesErrorKind = iota
	ProcessInlinesToken
	ProcessInlinesFirstRule
	ProcessInlinesNoReachableProductions
)

The kinds of error, in the order of upstream.

type ProdRef 

type ProdRef struct {
	Steps                []ProductionStep
	DynamicPrecedence    int32
	RequiresEOFLookahead bool
}

ProdRef is the contents of a production, with its steps taken from where they are stored.

ProdRef is ProdRef.

func (ProdRef) FirstSymbol 

func (p ProdRef) FirstSymbol() (Symbol, bool)

FirstSymbol returns the first symbol of the production, and false when the production is empty.

FirstSymbol is ProdRef::first_symbol.

type Production 

type Production struct {
	StepsStart        uint32
	StepsLen          uint32
	DynamicPrecedence int32
	// RequiresEOFLookahead is true when the production ends in eof(). Its
	// reduction happens only on the end of the input, and never as a shift.
	RequiresEOFLookahead bool
}

Production is a flat production: a range of steps, its dynamic precedence, and whether its reduction waits for the end of the input.

Production is Production.

func (Production) Contents added in v0.4.0 

func (p Production) Contents(steps []ProductionStep) ProdRef

Contents returns the contents of the production, with its steps taken from steps.

Contents is Production::contents.

func (Production) StepRange 

func (p Production) StepRange() (start, end int)

StepRange returns the range of the steps of the production.

StepRange is Production::step_range.

type ProductionInfo 

type ProductionInfo struct {
	AliasSequence []Alias
	FieldMap      FieldMap
}

ProductionInfo is the aliases and the fields of a production. A zero Alias in AliasSequence stands for the None of upstream, because no alias has the zero StrID.

ProductionInfo is ProductionInfo. The zero ProductionInfo is ProductionInfo::default.

func (*ProductionInfo) Equal 

func (p *ProductionInfo) Equal(other *ProductionInfo) bool

Equal reports whether two infos hold the same aliases and fields.

Equal is the PartialEq of ProductionInfo.

type ProductionInfoID 

type ProductionInfoID = uint32

ProductionInfoID is the index of a ProductionInfo in a parse table.

ProductionInfoID is ProductionInfoId.

type ProductionStep 

type ProductionStep struct {
	SymIndex uint32
	PrecVal  int32
	Alias    uint32
	Field    uint32
	Reserved uint16
	Flags    uint8
}

ProductionStep is one step of a production, packed as upstream packs it.

The bits of Flags are:

  • 0 to 2: the kind of the symbol, a SymbolType.
  • 3 and 4: the kind of the precedence: none 00, a number 01, a name 10.
  • 5 and 6: the associativity: none 00, left 01, right 10.
  • 7: whether the alias is named.

ProductionStep is ProductionStep.

func PackProductionStep 

func PackProductionStep(sym Symbol, prec Precedence, assoc Associativity, alias Alias, hasAlias bool, field StrID, reserved uint16) ProductionStep

PackProductionStep packs the parts of a production step. A zero field and a zero alias mean none.

PackProductionStep is ProductionStep::pack.

func (ProductionStep) Associativity 

func (s ProductionStep) Associativity() Associativity

Associativity returns the associativity of the step.

Associativity is ProductionStep::associativity.

func (ProductionStep) ChildType 

func (s ProductionStep) ChildType(defaultAliases AliasMap) ChildType

ChildType returns the child type of the step: its own alias, else the default alias of its symbol, else its symbol.

ChildType is ProductionStep::child_type in grammars.rs.

func (ProductionStep) GetAlias 

func (s ProductionStep) GetAlias() (Alias, bool)

GetAlias returns the alias of the step, and false when it has none.

GetAlias is ProductionStep::alias. The name differs because Alias is the field that holds the packed id.

func (ProductionStep) GetField 

func (s ProductionStep) GetField() (StrID, bool)

GetField returns the field name of the step, and false when it has none.

GetField is ProductionStep::field. The name differs because Field is the field that holds the packed id.

func (ProductionStep) NonTerminalIndex 

func (s ProductionStep) NonTerminalIndex() (NonTerminalIndex, bool)

NonTerminalIndex returns the index of the symbol, when it is a non-terminal.

NonTerminalIndex is ProductionStep::non_terminal_index.

func (ProductionStep) Precedence 

func (s ProductionStep) Precedence() Precedence

Precedence returns the precedence of the step.

Precedence is ProductionStep::precedence.

func (*ProductionStep) SetAlias 

func (s *ProductionStep) SetAlias(alias Alias, ok bool)

SetAlias sets the alias of the step, or removes it when ok is false.

SetAlias is ProductionStep::set_alias.

func (*ProductionStep) SetAliasNamed 

func (s *ProductionStep) SetAliasNamed(named bool)

SetAliasNamed sets whether the alias of the step is named.

SetAliasNamed is ProductionStep::set_alias_named.

func (*ProductionStep) SetAssociativity 

func (s *ProductionStep) SetAssociativity(assoc Associativity)

SetAssociativity sets the associativity of the step.

SetAssociativity is ProductionStep::set_associativity.

func (*ProductionStep) SetField 

func (s *ProductionStep) SetField(field StrID)

SetField sets the field name of the step. A zero id removes it.

SetField is ProductionStep::set_field.

func (*ProductionStep) SetPrecedence 

func (s *ProductionStep) SetPrecedence(prec Precedence)

SetPrecedence sets the precedence of the step.

SetPrecedence is ProductionStep::set_precedence.

func (ProductionStep) Symbol 

func (s ProductionStep) Symbol() Symbol

Symbol returns the symbol of the step.

Symbol is ProductionStep::symbol.

type ProductionStore 

type ProductionStore struct {
	Steps       []ProductionStep
	Productions []Production
	VarProds    [][2]uint32
}

ProductionStore is the flat output of flattening: one store of steps, each production as a range in it, and the productions of each variable.

ProductionStore is ProductionStore.

func (*ProductionStore) Production added in v0.4.0 

func (s *ProductionStore) Production(id uint32) ProdRef

Production returns the production at a position in Productions.

Production is ProductionStore::production.

type RegexError 

type RegexError struct {
	Pattern string
	Kind    RegexErrorKind
	// NestLimit is the limit of RegexNestLimitExceeded.
	NestLimit uint32
	// Other is the text of RegexOther.
	Other string
	// Span is the range that is wrong, or nil.
	Span *PatternSpan
	// AuxSpan is a second range, such as the first use of a duplicate
	// flag, or nil.
	AuxSpan *PatternSpan
}

RegexError is a pattern that parsePattern rejects, with the pattern and the range in it that is wrong.

RegexError is RegexError.

func (*RegexError) Error 

func (e *RegexError) Error() string

Error returns the text of the error: the pattern, a line of carets under the ranges that are wrong, and the kind.

Error is the Display of RegexError.

type RegexErrorKind 

type RegexErrorKind uint8

RegexErrorKind is the kind of an error of a pattern.

RegexErrorKind is RegexErrorKind, the mirror of the error kinds of regex-syntax. 

const (
	RegexCaptureLimitExceeded RegexErrorKind = iota
	RegexClassEscapeInvalid
	RegexClassRangeInvalid
	RegexClassRangeLiteral
	RegexClassUnclosed
	RegexDecimalEmpty
	RegexDecimalInvalid
	RegexEscapeHexEmpty
	RegexEscapeHexInvalid
	RegexEscapeHexInvalidDigit
	RegexEscapeUnexpectedEOF
	RegexEscapeUnrecognized
	RegexFlagDanglingNegation
	RegexFlagUnexpectedEOF
	RegexFlagUnrecognized
	RegexGroupNameEmpty
	RegexGroupNameInvalid
	RegexGroupNameUnexpectedEOF
	RegexGroupUnclosed
	RegexGroupUnopened
	RegexRepetitionCountInvalid
	RegexRepetitionCountDecimalEmpty
	RegexRepetitionCountUnclosed
	RegexRepetitionMissing
	RegexSpecialWordBoundaryUnclosed
	RegexSpecialWordBoundaryUnrecognized
	RegexSpecialWordOrRepetitionUnexpectedEOF
	RegexUnicodeClassInvalid
	RegexUnsupportedBackreference
	RegexUnsupportedLookAround
	RegexUnicodeNotAllowed
	RegexInvalidUTF8
	RegexInvalidLineTerminator
	RegexUnicodePropertyNotFound
	RegexUnicodePropertyValueNotFound
	RegexUnicodePerlClassNotFound
	RegexUnicodeCaseUnavailable
	RegexFlagDuplicate
	RegexFlagRepeatedNegation
	RegexGroupNameDuplicate
	RegexNestLimitExceeded
	// RegexOther is a kind that regex-syntax added after upstream wrote the
	// mirror.
	RegexOther
)

The kinds of error of a pattern, in the order of upstream: the kinds of the parser, the kinds of the translator, and then the kinds that hold data.

type RenderInput 

type RenderInput struct {
	// Name is the name of the grammar, in StrPool.
	Name           StrID
	Tables         *Tables
	SyntaxGrammar  *SyntaxGrammar
	LexicalGrammar *LexicalGrammar
	DefaultAliases AliasMap
	StrPool        *StrPool
	ABIVersion     int
	// SemanticVersion is the version of the grammar, or nil when it has
	// none.
	SemanticVersion    *SemanticVersion
	SupertypeSymbolMap SupertypeSymbolMap
}

RenderInput is what a backend reads to write a parser: the grammar and its tables, as the generator builds them (D8).

RenderInput holds the arguments of render_c_code.

type ReservedWordContext 

type ReservedWordContext struct {
	Name  StrID
	Roots []RuleID
}

ReservedWordContext is a named set of reserved words.

ReservedWordContext is ReservedWordContext.

type ReservedWordSetID 

type ReservedWordSetID uint32

ReservedWordSetID is the id of a set of reserved words.

ReservedWordSetID is ReservedWordSetId.

type Resolution 

type Resolution struct {
	Kind    ResolutionKind
	Symbols []string
}

Resolution is a change to the grammar that resolves a conflict: the kind of change, and the rules that it changes.

Resolution is Resolution, an enum with data upstream. Each variant has the field symbols.

func (Resolution) String 

func (r Resolution) String() string

String returns the text of the resolution.

String is the Display of Resolution.

type ResolutionKind 

type ResolutionKind uint8

ResolutionKind is the kind of a resolution of a conflict. 

const (
	ResolutionPrecedence ResolutionKind = iota
	ResolutionAssociativity
	ResolutionAddConflict
)

The kinds of resolution, in the order of upstream.

type Rule 

type Rule struct {
	Kind     RuleKind
	Str      StrID
	Flags    StrID
	Sym      Symbol
	Children RuleIDRange
	Child    RuleID
	Params   ParamsID
}

Rule is one node of a rule pool.

Rule is Rule, an enum with data upstream. The fields that a kind uses are:

  • RuleString and RuleNamedSymbol: Str.
  • RulePattern: Str for the pattern and Flags for its flags.
  • RuleSym: Sym.
  • RuleSeq and RuleChoice: Children.
  • RuleRepeat: Child.
  • RuleMetadata: Params and Child.
  • RuleReserved: Child, and Str for the context.

func (Rule) Symbol 

func (r Rule) Symbol() (Symbol, bool)

Symbol returns the symbol of a RuleSym node.

Symbol is Rule::symbol.

type RuleID 

type RuleID uint32

RuleID is the index of a node in a rule pool.

RuleID is RuleId.

func TrySeq 

func TrySeq[T any](p *RulePool, items []T, f func(*RulePool, T) (RuleID, error)) (RuleID, error)

TrySeq adds a sequence of the nodes that f makes from items. The children are reserved first, so that the nodes that f adds do not come between them. It stops at the first error of f.

TrySeq is RulePool::try_seq.

func (RuleID) Index 

func (id RuleID) Index() int

Index returns the index of the node.

Index is RuleId::index.

type RuleIDRange 

type RuleIDRange struct {
	Start uint32
	Len   uint32
}

RuleIDRange is the range [Start, Start+Len) of the children of a pool.

RuleIDRange is RuleIdRange.

type RuleKind 

type RuleKind uint8

RuleKind is the kind of a rule node. 

const (
	RuleBlank RuleKind = iota
	RuleString
	RulePattern
	RuleNamedSymbol
	RuleSym
	RuleSeq
	RuleChoice
	RuleRepeat
	RuleEOF
	RuleMetadata
	RuleReserved
)

The kinds of rule node, in the order of the enum Rule.

type RulePool 

type RulePool struct {
	// contains filtered or unexported fields
}

RulePool is the pool of the rule nodes, of their children and their metadata, and of the strings that they intern. It only grows. A pass rewrites nodes in place, and it can leave a subtree that nothing reaches.

RulePool is RulePool.

func NewRulePool 

func NewRulePool() *RulePool

NewRulePool returns an empty pool.

NewRulePool is RulePool::default.

func (*RulePool) Alias 

func (p *RulePool) Alias(content RuleID, value StrID, isNamed bool) RuleID

Alias gives content an alias.

Alias is RulePool::alias.

func (*RulePool) Blank 

func (p *RulePool) Blank() RuleID

Blank adds a blank node.

Blank is RulePool::blank.

func (*RulePool) ChildSlice 

func (p *RulePool) ChildSlice(r RuleIDRange) []RuleID

ChildSlice returns the children of a range.

ChildSlice is RulePool::child_slice.

func (*RulePool) Choice 

func (p *RulePool) Choice(ids []RuleID) RuleID

Choice adds a choice of nodes. It flattens nested choices and drops a node that is the same as one before it, and it keeps a Choice node even for one member.

Choice is RulePool::choice.

func (*RulePool) CollectReferencedIDs 

func (p *RulePool) CollectReferencedIDs(id RuleID, skipTopLevel bool, out *[]StrID)

CollectReferencedIDs appends to out the name of each named symbol that id reaches. When skipTopLevel is true, a named symbol at the root of id does not count, which an entry of externals needs, because it names itself.

CollectReferencedIDs is RulePool::collect_referenced_ids.

func (*RulePool) EOF 

func (p *RulePool) EOF() RuleID

EOF adds a node for the end of the input.

EOF is RulePool::eof.

func (*RulePool) Field 

func (p *RulePool) Field(name StrID, content RuleID) RuleID

Field gives content a field name.

Field is RulePool::field.

func (*RulePool) ImmediateToken 

func (p *RulePool) ImmediateToken(content RuleID) RuleID

ImmediateToken makes content a token that follows the last token with no extra between them.

ImmediateToken is RulePool::immediate_token.

func (*RulePool) Intern 

func (p *RulePool) Intern(s string) StrID

Intern interns a string in the pool.

Intern is RulePool::intern.

func (*RulePool) Interner 

func (p *RulePool) Interner() *StrPool

Interner returns the string pool.

Interner is RulePool::into_interner.

func (*RulePool) NamedSymbol 

func (p *RulePool) NamedSymbol(name StrID) RuleID

NamedSymbol adds a node that names a rule.

NamedSymbol is RulePool::named_symbol.

func (*RulePool) Node 

func (p *RulePool) Node(id RuleID) Rule

Node returns a node.

Node is RulePool::node.

func (*RulePool) NodeCount 

func (p *RulePool) NodeCount() int

NodeCount returns the number of nodes.

NodeCount is RulePool::node_count.

func (*RulePool) Params 

func (p *RulePool) Params(id ParamsID) MetadataParams

Params returns metadata.

Params is RulePool::params.

func (*RulePool) Pattern 

func (p *RulePool) Pattern(value, flags StrID) RuleID

Pattern adds a pattern node.

Pattern is RulePool::pattern.

func (*RulePool) Prec 

func (p *RulePool) Prec(value Precedence, content RuleID) RuleID

Prec gives content a precedence.

Prec is RulePool::prec.

func (*RulePool) PrecDynamic 

func (p *RulePool) PrecDynamic(value int32, content RuleID) RuleID

PrecDynamic gives content a dynamic precedence.

PrecDynamic is RulePool::prec_dynamic.

func (*RulePool) PrecLeft 

func (p *RulePool) PrecLeft(value Precedence, content RuleID) RuleID

PrecLeft gives content a precedence that associates to the left.

PrecLeft is RulePool::prec_left.

func (*RulePool) PrecRight 

func (p *RulePool) PrecRight(value Precedence, content RuleID) RuleID

PrecRight gives content a precedence that associates to the right.

PrecRight is RulePool::prec_right.

func (*RulePool) PushChildren 

func (p *RulePool) PushChildren(ids []RuleID) RuleIDRange

PushChildren adds children and returns their range.

PushChildren is RulePool::push_children.

func (*RulePool) PushNode 

func (p *RulePool) PushNode(node Rule) RuleID

PushNode adds a node and returns its id.

PushNode is RulePool::push_node.

func (*RulePool) PushParams 

func (p *RulePool) PushParams(params MetadataParams) ParamsID

PushParams adds metadata and returns its id.

PushParams is RulePool::push_params.

func (*RulePool) Repeat 

func (p *RulePool) Repeat(content RuleID) RuleID

Repeat adds a node that repeats content one or more times.

Repeat is RulePool::repeat.

func (*RulePool) Reserved 

func (p *RulePool) Reserved(content RuleID, ctx StrID) RuleID

Reserved adds a node that parses content with the reserved words of a context.

Reserved is RulePool::reserved.

func (*RulePool) Resolve 

func (p *RulePool) Resolve(id StrID) string

Resolve returns an interned string.

Resolve is RulePool::resolve.

func (*RulePool) RuleIsReferenced 

func (p *RulePool) RuleIsReferenced(id RuleID, target StrID, isExternal bool) bool

RuleIsReferenced reports whether a rule names target. It is how the generator finds whether a rule uses a variable. When isExternal is true, a named symbol at the top of the rule does not count: for an external rule and a normal rule that are both called foo, the external rule named foo does not use foo unless another rule inside it does.

RuleIsReferenced is RulePool::rule_is_referenced.

func (*RulePool) Seq 

func (p *RulePool) Seq(ids []RuleID) RuleID

Seq adds a sequence of nodes.

Seq is RulePool::seq.

func (*RulePool) SetNode 

func (p *RulePool) SetNode(id RuleID, node Rule)

SetNode replaces a node.

SetNode is RulePool::set_node.

func (*RulePool) SetParams 

func (p *RulePool) SetParams(id ParamsID, params MetadataParams)

SetParams replaces metadata.

SetParams is RulePool::set_params.

func (*RulePool) String 

func (p *RulePool) String(value StrID) RuleID

String adds a string node.

String is RulePool::string.

func (*RulePool) SubtreeEqual 

func (p *RulePool) SubtreeEqual(a, b RuleID) bool

SubtreeEqual reports whether two subtrees are the same.

SubtreeEqual is RulePool::subtree_eq.

func (*RulePool) SubtreeHash 

func (p *RulePool) SubtreeHash(root RuleID) uint64

SubtreeHash returns a hash of a subtree. Two subtrees that SubtreeEqual finds equal have the same hash.

SubtreeHash is RulePool::subtree_hash. Upstream hashes with FxHasher. The hash only finds candidates in a memo, and SubtreeEqual decides between them, so another hash gives the same result, and this one uses FNV.

func (*RulePool) SubtreeMatchesEmptyString 

func (p *RulePool) SubtreeMatchesEmptyString(id RuleID) bool

SubtreeMatchesEmptyString reports whether the subtree at id can only match the empty string.

SubtreeMatchesEmptyString is RulePool::subtree_matches_empty_str.

func (*RulePool) Token 

func (p *RulePool) Token(content RuleID) RuleID

Token makes content a token.

Token is RulePool::token.

type SemanticVersion 

type SemanticVersion struct {
	Major, Minor, Patch uint8
}

SemanticVersion is the version of a grammar, from tree-sitter.json.

SemanticVersion is the (u8, u8, u8) of upstream.

func ReadGrammarVersion 

func ReadGrammarVersion(repoPath string) (*SemanticVersion, error)

ReadGrammarVersion reads the version of a grammar from the nearest tree-sitter.json, in the folder of the grammar or in a folder above it. It returns nil when no folder has one. Each number is cut to 8 bits, as the cast of upstream does.

ReadGrammarVersion is read_grammar_version. The text of an error of JSON is the text of encoding/json, and the text of a version error is that of the port, not of the Rust crate semver.

type SplitCriterion added in v0.4.0 

type SplitCriterion[S any] interface {
	// ShouldSplit reports whether left and right, from the same group, must
	// be in different groups. This must be symmetric.
	//
	// ShouldSplit is SplitCriterion::should_split.
	ShouldSplit(left, right *S, groupIDsByStateID []uint32) bool
	// Signature returns a hash of what Equivalent compares, and false if
	// this criterion does not sort states into classes.
	//
	// Signature is SplitCriterion::signature.
	Signature(state *S, groupIDsByStateID []uint32) (uint64, bool)
	// Equivalent reports whether two states with the same signature are
	// equivalent.
	//
	// Equivalent is SplitCriterion::equivalent.
	Equivalent(left, right *S, groupIDsByStateID []uint32) bool
	// StartGroup is called before a group's states are compared, e.g. to
	// reset what CompatibleWithAll keeps.
	//
	// StartGroup is SplitCriterion::start_group.
	StartGroup()
	// CompatibleWithAll reports whether state can stay in its group with all
	// of kept.
	//
	// kept holds the first state of each class that stayed before state, in
	// order. This is only asked once ShouldSplit found that state can stay
	// with kept[0]. false is always correct: the states are then compared one
	// at a time.
	//
	// CompatibleWithAll is SplitCriterion::compatible_with_all.
	CompatibleWithAll(state *S, kept []uint32, groupIDsByStateID []uint32) bool
}

SplitCriterion decides which states of a group SplitStateIDGroups separates.

Besides ShouldSplit, a criterion can sort a group's states into classes of equivalent states. These are states that never need to be split from each other, and that need to be split from exactly the same states. Only the first state of each class is then compared, and the rest of the class follows it.

SplitCriterion is the trait SplitCriterion. Each type that implements it writes the default methods of the trait that it keeps.

type StrID 

type StrID uint32

StrID is the id of an interned string, a 1-based index into the string table of a pool.

StrID is StrId. 

const (
	// EmptyStrID is the id of "".
	EmptyStrID StrID = 1
	// EndNameID is the id of "end".
	EndNameID StrID = 2
)

The ids that every pool starts with.

func StrIDFromRaw 

func StrIDFromRaw(raw uint32) StrID

StrIDFromRaw is the inverse of Raw. It panics when raw is 0, which no pool hands out.

StrIDFromRaw is StrId::from_raw.

func (StrID) Index 

func (id StrID) Index() int

Index returns the 0-based index of the id.

Index is StrId::index.

func (StrID) Raw 

func (id StrID) Raw() uint32

Raw returns the 1-based id, for packed encodings where 0 means none.

Raw is StrId::raw.

type StrPool 

type StrPool struct {
	// contains filtered or unexported fields
}

StrPool is an append-only pool of strings. The zero StrPool is not ready to use. Make one with NewStrPool.

StrPool is StrPool. Upstream finds a string with a hash table of ids. Go uses a map from the string to its id, which gives the same ids, because an id depends only on the order in which strings are interned.

func NewStrPool 

func NewStrPool() *StrPool

NewStrPool returns a pool that holds "" and "end", with the ids EmptyStrID and EndNameID.

NewStrPool is StrPool::default.

func (*StrPool) Clone 

func (p *StrPool) Clone() *StrPool

Clone returns a copy of the pool.

Clone is the Clone of StrPool.

func (*StrPool) Intern 

func (p *StrPool) Intern(s string) StrID

Intern returns the id of a string, and adds the string to the pool when it is new.

Intern is StrPool::intern.

func (*StrPool) Resolve 

func (p *StrPool) Resolve(id StrID) string

Resolve returns the string of an id.

Resolve is StrPool::resolve.

type SuperTypeCycleError 

type SuperTypeCycleError struct {
	Items []string
}

SuperTypeCycleError is the error of supertypes that contain each other in a cycle.

SuperTypeCycleError is SuperTypeCycleError.

func (*SuperTypeCycleError) Error 

func (e *SuperTypeCycleError) Error() string

Error returns the text of the error.

Error is the Display of SuperTypeCycleError.

type SupertypeAliasCollisionError 

type SupertypeAliasCollisionError struct {
	Name string
}

SupertypeAliasCollisionError is the error of a named alias that has the name of a supertype.

SupertypeAliasCollisionError is VariableInfoError::SupertypeAliasCollision.

func (*SupertypeAliasCollisionError) Error 

func (e *SupertypeAliasCollisionError) Error() string

Error returns the text of the error.

type SupertypeSymbolMap 

type SupertypeSymbolMap map[Symbol][]ChildType

SupertypeSymbolMap maps each supertype to the types of its visible children.

SupertypeSymbolMap is the BTreeMap<Symbol, Vec<ChildType>> that get_supertype_symbol_map returns. It iterates in the order of the symbols, and Sorted gives that order.

func GetSupertypeSymbolMap 

func GetSupertypeSymbolMap(syntaxGrammar *SyntaxGrammar, defaultAliases AliasMap, variableInfo []VariableInfo) SupertypeSymbolMap

GetSupertypeSymbolMap returns the child types of each supertype.

Upstream also builds a map from each alias to its symbols here, and never reads it. Go rejects a variable that is not read, and the map changes nothing, so the port leaves it out.

GetSupertypeSymbolMap is get_supertype_symbol_map.

func (SupertypeSymbolMap) Sorted 

func (m SupertypeSymbolMap) Sorted() iter.Seq2[Symbol, []ChildType]

Sorted returns each supertype and its child types, in the order of the symbols.

type Symbol 

type Symbol struct {
	// contains filtered or unexported fields
}

Symbol is a symbol of a grammar: a kind, and the index in the table of that kind.

Symbol is Symbol. Two symbols compare with ==, and CompareSymbol orders them by kind and then by index.

func ExternalSymbol 

func ExternalSymbol(index int) Symbol

ExternalSymbol returns the external token with an index.

ExternalSymbol is Symbol::external.

func NonTerminalSymbol 

func NonTerminalSymbol(index int) Symbol

NonTerminalSymbol returns the non-terminal with an index.

NonTerminalSymbol is Symbol::non_terminal.

func TerminalSymbol 

func TerminalSymbol(index int) Symbol

TerminalSymbol returns the terminal with an index.

TerminalSymbol is Symbol::terminal.

func (Symbol) ExternalIndex 

func (s Symbol) ExternalIndex() (ExternalTokenIndex, bool)

ExternalIndex returns the index of an external token.

ExternalIndex is Symbol::external_index.

func (Symbol) Kind 

func (s Symbol) Kind() SymbolType

Kind returns the kind of the symbol.

Kind is Symbol::kind.

func (Symbol) NonTerminalIndex 

func (s Symbol) NonTerminalIndex() (NonTerminalIndex, bool)

NonTerminalIndex returns the index of a non-terminal.

NonTerminalIndex is Symbol::non_terminal_index.

func (Symbol) TerminalIndex 

func (s Symbol) TerminalIndex() (TerminalIndex, bool)

TerminalIndex returns the index of a terminal.

TerminalIndex is Symbol::terminal_index.

type SymbolType 

type SymbolType uint8

SymbolType is the kind of a symbol. The order of the values is the order of upstream, and symbols sort by it.

SymbolType is SymbolType. 

const (
	SymbolExternal SymbolType = iota
	SymbolEnd
	SymbolEndOfNonTerminalExtra
	SymbolTerminal
	SymbolNonTerminal
)

The kinds of symbol.

type SyntaxGrammar 

type SyntaxGrammar struct {
	Variables           []SyntaxVariable
	ExtraSymbols        []Symbol
	ExpectedConflicts   [][]Symbol
	ExternalTokens      []ExternalToken
	SupertypeSymbols    []Symbol
	VariablesToInline   []Symbol
	WordToken           Symbol
	HasWordToken        bool
	PrecedenceOrderings [][]PrecedenceEntry
	ReservedWordSets    []TokenSet

	Steps       []ProductionStep
	Productions []Production
	VarProds    [][2]uint32
}

SyntaxGrammar is the grammar of the parser, after the tokens are extracted and the rules are flattened.

SyntaxGrammar is SyntaxGrammar.

func (*SyntaxGrammar) Production 

func (g *SyntaxGrammar) Production(id uint32) ProdRef

Production returns a production by its id.

Production is SyntaxGrammar::production.

func (*SyntaxGrammar) VariableProdIDs 

func (g *SyntaxGrammar) VariableProdIDs(variableIndex int) (start, end uint32)

VariableProdIDs returns the range of the ids of the productions of a variable.

VariableProdIDs is SyntaxGrammar::variable_prod_ids.

type SyntaxVariable 

type SyntaxVariable struct {
	Name StrID
	Kind VariableType
}

SyntaxVariable is a non-terminal of the syntax grammar.

SyntaxVariable is SyntaxVariable.

func (SyntaxVariable) IsAuxiliary 

func (v SyntaxVariable) IsAuxiliary() bool

IsAuxiliary reports whether the generator made the variable.

IsAuxiliary is SyntaxVariable::is_auxiliary.

func (SyntaxVariable) IsHidden 

func (v SyntaxVariable) IsHidden() bool

IsHidden reports whether the variable is hidden.

IsHidden is SyntaxVariable::is_hidden.

type Tables 

type Tables struct {
	ParseTable         ParseTable
	MainLexTable       LexTable
	KeywordLexTable    LexTable
	LargeCharacterSets []LargeCharacterSet
}

Tables is the parse table and the lexer tables of a grammar.

Tables is Tables.

func BuildTables 

func BuildTables(syntaxGrammar *SyntaxGrammar, lexicalGrammar *LexicalGrammar, simpleAliases AliasMap, variableInfo []VariableInfo, inlines *InlinedProductionMap, strPool *StrPool, optimizations OptLevel, diagnostics *[]Diagnostic) (*Tables, error)

BuildTables builds the parse table and the lexer tables of a prepared grammar.

BuildTables is build_tables. The port leaves out its argument report_symbol_name, which only report_state_info reads.

type TerminalEntries added in v0.4.0 

type TerminalEntries = ParseStateEntries[ActionListID]

TerminalEntries holds the terminal entries of a parse state: the action list for each lookahead.

TerminalEntries is TerminalEntries.

type TerminalIndex 

type TerminalIndex uint32

TerminalIndex is the index of a terminal. It is a type of the macro symbol_index.

func (TerminalIndex) Symbol 

func (i TerminalIndex) Symbol() Symbol

Symbol returns the symbol of the index.

type TokenConflictMap 

type TokenConflictMap struct {
	// contains filtered or unexported fields
}

TokenConflictMap holds the kinds of overlap between each pair of tokens of a lexical grammar.

TokenConflictMap is TokenConflictMap.

func NewTokenConflictMap 

func NewTokenConflictMap(grammar *LexicalGrammar, followingTokens []TokenSet) *TokenConflictMap

NewTokenConflictMap creates a token conflict map from a lexical grammar, which describes the structure of each token, and followingTokens, which holds for each token the tokens that can come immediately after it.

It analyzes the possible kinds of overlap between each pair of tokens and stores them in a matrix.

NewTokenConflictMap is TokenConflictMap::new.

func (*TokenConflictMap) DoesConflict 

func (m *TokenConflictMap) DoesConflict(i, j int) bool

DoesConflict reports whether the status of the pair (i, j) holds DOES_MATCH_VALID_CONT, DOES_MATCH_SEPARATORS or MATCHES_SAME_STRING.

DoesConflict is TokenConflictMap::does_conflict. Upstream reads the matrix without a bounds check. Go checks the bounds, and an index out of bounds panics.

func (*TokenConflictMap) DoesMatchDifferentString 

func (m *TokenConflictMap) DoesMatchDifferentString(i, j int) bool

DoesMatchDifferentString reports whether token i matches any strings that token j does not match.

DoesMatchDifferentString is TokenConflictMap::does_match_different_string.

func (*TokenConflictMap) DoesMatchPrefix 

func (m *TokenConflictMap) DoesMatchPrefix(i, j int) bool

DoesMatchPrefix reports whether token i matches any strings that are prefixes of strings that token j matches.

DoesMatchPrefix is TokenConflictMap::does_match_prefix.

func (*TokenConflictMap) DoesMatchSameString 

func (m *TokenConflictMap) DoesMatchSameString(i, j int) bool

DoesMatchSameString reports whether token i matches any strings that token j also matches, where token i is preferred over token j.

DoesMatchSameString is TokenConflictMap::does_match_same_string.

func (*TokenConflictMap) DoesMatchShorterOrLonger 

func (m *TokenConflictMap) DoesMatchShorterOrLonger(i, j int) bool

DoesMatchShorterOrLonger reports whether the status of the pair (i, j) holds DOES_MATCH_VALID_CONT or DOES_MATCH_SEPARATORS, and the status of the pair (j, i) does not hold DOES_MATCH_SEPARATORS.

DoesMatchShorterOrLonger is TokenConflictMap::does_match_shorter_or_longer.

func (*TokenConflictMap) DoesOverlap 

func (m *TokenConflictMap) DoesOverlap(i, j int) bool

DoesOverlap reports whether token i overlaps token j in any way.

DoesOverlap is TokenConflictMap::does_overlap.

func (*TokenConflictMap) HasSameConflictStatus 

func (m *TokenConflictMap) HasSameConflictStatus(a, b, other int) bool

HasSameConflictStatus reports whether tokens a and b have the same conflict status with token other.

HasSameConflictStatus is TokenConflictMap::has_same_conflict_status.

type TokenSet 

type TokenSet struct {
	// contains filtered or unexported fields
}

TokenSet is a set of tokens, as bit vectors. A token is a small number, of about 400 at most, so a bit for each token holds a set well.

TokenSet is TokenSet.

func NewTokenSetWithCapacity 

func NewTokenSetWithCapacity(terminals, externals int) TokenSet

NewTokenSetWithCapacity returns an empty set with room for the tokens.

NewTokenSetWithCapacity is TokenSet::with_capacity.

func TokenSetFrom 

func TokenSetFrom(symbols iter.Seq[Symbol]) TokenSet

TokenSetFrom returns the set of the symbols.

TokenSetFrom is the FromIterator of TokenSet.

func (*TokenSet) All 

func (s *TokenSet) All() iter.Seq[Symbol]

All returns each symbol in the set: the terminals, then the external tokens, then the end of the input, then the end of a non-terminal extra.

All is TokenSet::iter.

func (*TokenSet) Clone 

func (s *TokenSet) Clone() TokenSet

Clone returns a copy of the set.

Clone is the Clone of TokenSet.

func (*TokenSet) Compare 

func (s *TokenSet) Compare(other *TokenSet) int

Compare orders two sets by their terminals, then their external tokens, then the end of the input, then the end of a non-terminal extra.

Compare is the Ord of TokenSet.

func (*TokenSet) Contains 

func (s *TokenSet) Contains(sym Symbol) bool

Contains reports whether the set holds a symbol. It panics for a non-terminal, which a token set never holds.

Contains is TokenSet::contains.

func (*TokenSet) ContainsTerminal 

func (s *TokenSet) ContainsTerminal(index int) bool

ContainsTerminal reports whether the set holds the terminal with an index.

ContainsTerminal is TokenSet::contains_terminal.

func (*TokenSet) Equal 

func (s *TokenSet) Equal(other *TokenSet) bool

Equal reports whether two sets hold the same symbols.

Equal is the PartialEq of TokenSet.

func (*TokenSet) Externals 

func (s *TokenSet) Externals() iter.Seq[ExternalTokenIndex]

Externals returns the index of each external token in the set.

Externals is TokenSet::externals.

func (*TokenSet) Insert 

func (s *TokenSet) Insert(sym Symbol)

Insert adds a symbol to the set. It panics for a non-terminal.

Insert is TokenSet::insert.

func (*TokenSet) InsertAll 

func (s *TokenSet) InsertAll(other *TokenSet) bool

InsertAll adds every symbol of other, and reports whether any was new.

InsertAll is TokenSet::insert_all.

func (*TokenSet) InsertAllTerminals 

func (s *TokenSet) InsertAllTerminals(other *TokenSet) bool

InsertAllTerminals adds the terminals of other, and reports whether any was new.

InsertAllTerminals is TokenSet::insert_all_terminals.

func (*TokenSet) IsEmpty 

func (s *TokenSet) IsEmpty() bool

IsEmpty reports whether the set holds no symbol.

IsEmpty is TokenSet::is_empty.

func (*TokenSet) Key 

func (s *TokenSet) Key() string

Key returns a string that is the same for two equal sets and differs for two sets that are not equal, for a map key. It stands in for the Hash and the Eq of TokenSet. The key starts with the length of the terminal part, because the bytes of a part can hold any value, so no separator byte can mark where the part ends.

func (*TokenSet) Len 

func (s *TokenSet) Len() int

Len returns the number of symbols in the set.

Len is TokenSet::len.

func (*TokenSet) Remove 

func (s *TokenSet) Remove(sym Symbol) bool

Remove removes a symbol from the set, and reports whether the set held it. It panics for a non-terminal.

Remove is TokenSet::remove.

func (*TokenSet) TerminalWords 

func (s *TokenSet) TerminalWords() []uint64

TerminalWords returns the words of the terminal bits.

TerminalWords is TokenSet::terminal_bits_words.

func (*TokenSet) Terminals 

func (s *TokenSet) Terminals() iter.Seq[TerminalIndex]

Terminals returns the index of each terminal in the set.

Terminals is TokenSet::terminals.

type TokenSetDisplay 

type TokenSetDisplay struct {
	Set     *TokenSet
	Syntax  *SyntaxGrammar
	Lexical *LexicalGrammar
	Pool    *StrPool
}

TokenSetDisplay shows a token set as text, for the messages of the generator.

TokenSetDisplay is TokenSetDisplay.

func (TokenSetDisplay) String 

func (d TokenSetDisplay) String() string

String returns the text of the token set.

String is the Display of TokenSetDisplay.

type UndeclaredPrecedenceError 

type UndeclaredPrecedenceError struct {
	Precedence string
	Rule       string
}

UndeclaredPrecedenceError is the error of a named precedence that no list in precedences declares.

UndeclaredPrecedenceError is UndeclaredPrecedenceError.

func (*UndeclaredPrecedenceError) Error 

func (e *UndeclaredPrecedenceError) Error() string

Error returns the text of the error.

type Variable 

type Variable struct {
	Name StrID
	Root RuleID
}

Variable is a rule of a grammar: its name, and the root of its rule in the pool.

Variable is Variable.

type VariableInfo 

type VariableInfo struct {
	Fields                 map[StrID]*FieldInfo
	Children               FieldInfo
	ChildrenWithoutFields  FieldInfo
	HasMultiStepProduction bool
}

VariableInfo is the summary of the public structure of one variable of the syntax grammar: its fields, its visible children, and its named children without a field.

Fields is a Go map, and a loop over it has a random order. Upstream keeps it in an FxHashMap, and no caller of upstream lets the order of that map reach the output. A caller that needs an order must sort the keys.

VariableInfo is VariableInfo.

func GetVariableInfo 

func GetVariableInfo(syntaxGrammar *SyntaxGrammar, lexicalGrammar *LexicalGrammar, defaultAliases AliasMap, strPool *StrPool) ([]VariableInfo, error)

GetVariableInfo computes a summary of the public structure of each variable in the grammar. Each variable in the grammar is a distinct public node type. The result has one VariableInfo for each variable, by index.

The information about each node type N is:

  1. Children: the types of visible children that can appear in N.
  2. Fields: the fields that N can have. For each field, the types of visible children that the field can hold, whether N always has the field, and whether N can have several children in the field.
  3. ChildrenWithoutFields: the other named children of N, which have no field. Their types, whether N always has at least one of them, and whether N can have several of them.

Each summary accounts for two indirect factors:

  1. Hidden nodes. When a parent node N has a hidden child C, the visible children of C seem to be direct children of N.
  2. Aliases. When a parent node type M has the alias N, a node that seems to have the type N can have the inner structure of M.

The error is an *InvalidSupertypeError or a *SupertypeAliasCollisionError.

GetVariableInfo is get_variable_info.

type VariableType 

type VariableType uint8

VariableType is the kind of a variable. The order of the values is the order of upstream.

VariableType is VariableType. 

const (
	VariableHidden VariableType = iota
	VariableAuxiliary
	VariableAnonymous
	VariableNamed
)

The kinds of variable.

func (VariableType) IsVisible 

func (t VariableType) IsVisible() bool

IsVisible reports whether a node of the kind shows in the tree.

IsVisible is VariableType::is_visible.

Source Files

View all Source files

Directories

Path Synopsis
backend
c
Package c is the C backend of the transit generator (D8).
 Package c is the C backend of the transit generator (D8).
go
Package golang is the Go backend of the transit generator (D8).
 Package golang is the Go backend of the transit generator (D8).
internal
fxhash
Package fxhash gives the generator the hash and the order of iteration of the hash sets that upstream tree-sitter uses.
 Package fxhash gives the generator the hash and the order of iteration of the hash sets that upstream tree-sitter uses.
regexsyntax/ast
Package ast is the syntax tree of a regular expression, its parser and a walk of the tree.
 Package ast is the syntax tree of a regular expression, its parser and a walk of the tree.
regexsyntax/hir
Package hir is the high level intermediate form (the HIR) of a regular expression, and the translator that makes it from a syntax tree.
 Package hir is the high level intermediate form (the HIR) of a regular expression, and the translator that makes it from a syntax tree.
regexsyntax/unicodetables
Package unicodetables holds the Unicode tables of the Rust crate regex-syntax 0.8.11, as Go data.
 Package unicodetables holds the Unicode tables of the Rust crate regex-syntax 0.8.11, as Go data.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.