v0.20.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Apr 17, 2023 License: Apache-2.0 Imports: 4 Imported by: 130



Package ast provides AST nodes and ancillary structures and algorithms.

Package ast provides AST nodes and ancillary structures and algorithms.



This section is empty.


View Source
var BopMap = map[string]BinaryOp{
	"*": BopMult,
	"/": BopDiv,
	"%": BopPercent,

	"+": BopPlus,
	"-": BopMinus,

	"<<": BopShiftL,
	">>": BopShiftR,

	">":  BopGreater,
	">=": BopGreaterEq,
	"<":  BopLess,
	"<=": BopLessEq,
	"in": BopIn,

	"==": BopManifestEqual,
	"!=": BopManifestUnequal,

	"&": BopBitwiseAnd,
	"^": BopBitwiseXor,
	"|": BopBitwiseOr,

	"&&": BopAnd,
	"||": BopOr,

BopMap is a map from binary operator token strings to BinaryOp values.

View Source
var UopMap = map[string]UnaryOp{
	"!": UopNot,
	"~": UopBitwiseNot,
	"+": UopPlus,
	"-": UopMinus,

UopMap is a map from unary operator token strings to UnaryOp values.


func FodderAppend added in v0.11.2

func FodderAppend(a *Fodder, elem FodderElement)

FodderAppend appends to the fodder but preserves constraints.

See FodderConcat below.

func FodderCountNewlines added in v0.11.2

func FodderCountNewlines(fodder Fodder) int

FodderCountNewlines returns the number of new line chars represented by the fodder.

func FodderElementCountNewlines added in v0.11.2

func FodderElementCountNewlines(elem FodderElement) int

FodderElementCountNewlines returns the number of new line chars represented by the fodder element

func FodderEnsureCleanNewline added in v0.11.2

func FodderEnsureCleanNewline(fodder *Fodder)

FodderEnsureCleanNewline adds a LineEnd to the fodder if necessary.

func FodderHasCleanEndline added in v0.11.2

func FodderHasCleanEndline(fodder Fodder) bool

FodderHasCleanEndline is true if the fodder doesn't end with an interstitial.

func FodderMoveFront added in v0.11.2

func FodderMoveFront(a *Fodder, b *Fodder)

FodderMoveFront moves b to the front of a.

func LocationBefore added in v0.16.0

func LocationBefore(a Location, b Location) bool

LocationBefore returns whether one code location refers to the location closer to the beginning of the file than the other one.


type Apply

type Apply struct {
	Target           Node
	FodderLeft       Fodder
	Arguments        Arguments
	FodderRight      Fodder
	TailStrictFodder Fodder
	// Always false if there were no arguments.
	TrailingComma bool
	TailStrict    bool

Apply represents a function call

type ApplyBrace

type ApplyBrace struct {
	Left  Node
	Right Node

ApplyBrace represents e { }. Desugared to e + { }.

type Arguments

type Arguments struct {
	Positional []CommaSeparatedExpr
	Named      []NamedArgument

Arguments represents positional and named arguments to a function call f(x, y, z=1).

type Array

type Array struct {
	Elements    []CommaSeparatedExpr
	CloseFodder Fodder
	// Always false if there were no elements.
	TrailingComma bool

Array represents array constructors [1, 2, 3].

type ArrayComp

type ArrayComp struct {
	Body                Node
	TrailingCommaFodder Fodder
	Spec                ForSpec
	CloseFodder         Fodder
	TrailingComma bool

ArrayComp represents array comprehensions (which are like Python list comprehensions)

type Assert

type Assert struct {
	Cond            Node
	Message         Node
	Rest            Node
	ColonFodder     Fodder
	SemicolonFodder Fodder

Assert represents an assert expression (not an object-level assert).

After parsing, message can be nil indicating that no message was specified. This AST is elimiated by desugaring.

type Binary

type Binary struct {
	Right    Node
	Left     Node
	OpFodder Fodder
	Op BinaryOp

Binary represents binary operators.

type BinaryOp

type BinaryOp int

BinaryOp represents a binary operator.

const (
	BopMult BinaryOp = iota







Binary operators

func (BinaryOp) String

func (b BinaryOp) String() string

type CommaSeparatedExpr added in v0.15.0

type CommaSeparatedExpr struct {
	Expr        Node
	CommaFodder Fodder

CommaSeparatedExpr represents an expression that is an element of a comma-separated list of expressions (e.g. in an array or the arguments of a call)

type CommaSeparatedID added in v0.15.0

type CommaSeparatedID struct {
	NameFodder  Fodder
	Name        Identifier
	CommaFodder Fodder

CommaSeparatedID represents an expression that is an element of a comma-separated list of identifiers (e.g. an array of parameters)

type Conditional

type Conditional struct {
	Cond        Node
	BranchTrue  Node
	BranchFalse Node
	ThenFodder  Fodder
	ElseFodder  Fodder

Conditional represents if/then/else.

After parsing, branchFalse can be nil indicating that no else branch was specified. The desugarer fills this in with a LiteralNull

type Context

type Context *string

Context represents the surrounding context of a node (e.g. a function it's in)

type DesugaredObject

type DesugaredObject struct {
	Asserts Nodes
	Fields  DesugaredObjectFields
	Locals  LocalBinds

DesugaredObject represents object constructors { f: e ... } after desugaring.

The assertions either return true or raise an error.

type DesugaredObjectField

type DesugaredObjectField struct {
	Name      Node
	Body      Node
	LocRange  LocationRange
	Hide      ObjectFieldHide
	PlusSuper bool

DesugaredObjectField represents a desugared object field.

type DesugaredObjectFields

type DesugaredObjectFields []DesugaredObjectField

DesugaredObjectFields represents a DesugaredObjectField slice.

type DiagnosticFileName added in v0.17.0

type DiagnosticFileName string

DiagnosticFileName is a file name used for diagnostics. It might be a dummy value, such as <std> or <extvar:something>. It should never be passed to an importer.

type Dollar

type Dollar struct{ NodeBase }

Dollar represents the $ keyword

type Error

type Error struct {
	Expr Node

Error represents the error e.

type Fodder added in v0.11.2

type Fodder []FodderElement

Fodder is stuff that is usually thrown away by lexers/preprocessors but is kept so that the source can be round tripped with near full fidelity.

func FodderConcat added in v0.11.2

func FodderConcat(a Fodder, b Fodder) Fodder

FodderConcat concats the two fodders but also preserves constraints.

Namely, a FodderLineEnd is not allowed to follow a FodderParagraph or a FodderLineEnd.

type FodderElement added in v0.11.2

type FodderElement struct {
	Comment []string
	Kind    FodderKind
	Blanks  int
	Indent  int

FodderElement is a single piece of fodder.

func MakeFodderElement added in v0.11.2

func MakeFodderElement(kind FodderKind, blanks int, indent int, comment []string) FodderElement

MakeFodderElement is a helper function that checks some preconditions.

type FodderKind added in v0.11.2

type FodderKind int

FodderKind is an enum.

const (
	// FodderLineEnd represents a line ending.
	// It indicates that the next token, paragraph, or interstitial
	// should be on a new line.
	// A single comment string is allowed, which flows before the new line.
	// The LineEnd fodder specifies the indentation level and vertical spacing
	// before whatever comes next.
	FodderLineEnd FodderKind = iota

	// FodderInterstitial represents a comment in middle of a line.
	// They must be /* C-style */ comments.
	// If it follows a token (i.e., it is the first fodder element) then it
	// appears after the token on the same line.  If it follows another
	// interstitial, it will also flow after it on the same line.  If it follows
	// a new line or a paragraph, it is the first thing on the following line,
	// after the blank lines and indentation specified by the previous fodder.
	// There is exactly one comment string.

	// FodderParagraph represents a comment consisting of at least one line.
	// // and # style comments have exactly one line.  C-style comments can have
	// more than one line.
	// All lines of the comment are indented according to the indentation level
	// of the previous new line / paragraph fodder.
	// The Paragraph fodder specifies the indentation level and vertical spacing
	// before whatever comes next.

type ForSpec

type ForSpec struct {
	ForFodder  Fodder
	VarFodder  Fodder
	Conditions []IfSpec
	Outer      *ForSpec
	Expr       Node
	VarName    Identifier
	InFodder   Fodder

ForSpec represents a for-specification in a comprehension. Example: expr for x in arr1 for y in arr2 for z in arr3 The order is the same as in python, i.e. the leftmost is the outermost.

Our internal representation reflects how they are semantically nested: ForSpec(z, outer=ForSpec(y, outer=ForSpec(x, outer=nil))) Any ifspecs are attached to the relevant ForSpec.

Ifs are attached to the one on the left, for example: expr for x in arr1 for y in arr2 if x % 2 == 0 for z in arr3 The if is attached to the y forspec.

It desugares to:

flatMap(\x ->
        flatMap(\y ->
                flatMap(\z -> [expr], arr3)

type Function

type Function struct {
	ParenLeftFodder  Fodder
	ParenRightFodder Fodder
	Body             Node
	Parameters       []Parameter
	// Always false if there were no parameters.
	TrailingComma bool

Function represents a function definition

type Identifier

type Identifier string

Identifier represents a variable / parameter / field name. +gen set

type IdentifierSet

type IdentifierSet map[Identifier]struct{}

IdentifierSet is the primary type that represents a set

func NewIdentifierSet

func NewIdentifierSet(a ...Identifier) IdentifierSet

NewIdentifierSet creates and returns a reference to an empty set.

func (IdentifierSet) Add

func (set IdentifierSet) Add(i Identifier) bool

Add adds an item to the current set if it doesn't already exist in the set.

func (IdentifierSet) AddIdentifiers

func (i IdentifierSet) AddIdentifiers(idents Identifiers)

AddIdentifiers adds a slice of identifiers to an identifier set.

func (IdentifierSet) Cardinality

func (set IdentifierSet) Cardinality() int

Cardinality returns how many items are currently in the set.

func (*IdentifierSet) Clear

func (set *IdentifierSet) Clear()

Clear clears the entire set to be the empty set.

func (IdentifierSet) Clone

func (set IdentifierSet) Clone() IdentifierSet

Clone returns a clone of the set. Does NOT clone the underlying elements.

func (IdentifierSet) Contains

func (set IdentifierSet) Contains(i Identifier) bool

Contains determines if a given item is already in the set.

func (IdentifierSet) ContainsAll

func (set IdentifierSet) ContainsAll(i ...Identifier) bool

ContainsAll determines if the given items are all in the set

func (IdentifierSet) Difference

func (set IdentifierSet) Difference(other IdentifierSet) IdentifierSet

Difference returns a new set with items in the current set but not in the other set

func (IdentifierSet) Equal

func (set IdentifierSet) Equal(other IdentifierSet) bool

Equal determines if two sets are equal to each other. If they both are the same size and have the same items they are considered equal. Order of items is not relevent for sets to be equal.

func (IdentifierSet) Intersect

func (set IdentifierSet) Intersect(other IdentifierSet) IdentifierSet

Intersect returns a new set with items that exist only in both sets.

func (IdentifierSet) IsSubset

func (set IdentifierSet) IsSubset(other IdentifierSet) bool

IsSubset determines if every item in the other set is in this set.

func (IdentifierSet) IsSuperset

func (set IdentifierSet) IsSuperset(other IdentifierSet) bool

IsSuperset determines if every item of this set is in the other set.

func (IdentifierSet) Iter

func (set IdentifierSet) Iter() <-chan Identifier

Iter returns a channel of type identifier that you can range over.

func (IdentifierSet) Remove

func (set IdentifierSet) Remove(i Identifier)

Remove allows the removal of a single item in the set.

func (IdentifierSet) SymmetricDifference

func (set IdentifierSet) SymmetricDifference(other IdentifierSet) IdentifierSet

SymmetricDifference returns a new set with items in the current set or the other set but not in both.

func (IdentifierSet) ToOrderedSlice

func (i IdentifierSet) ToOrderedSlice() []Identifier

ToOrderedSlice returns the elements of the current set as an ordered slice.

func (IdentifierSet) ToSlice

func (set IdentifierSet) ToSlice() []Identifier

ToSlice returns the elements of the current set as a slice

func (IdentifierSet) Union

func (set IdentifierSet) Union(other IdentifierSet) IdentifierSet

Union returns a new set with all items in both sets.

type Identifiers

type Identifiers []Identifier

Identifiers represents an Identifier slice.

type IfSpec

type IfSpec struct {
	Expr     Node
	IfFodder Fodder

IfSpec represents an if-specification in a comprehension.

type Import

type Import struct {
	File *LiteralString

Import represents import "file".

type ImportBin added in v0.19.0

type ImportBin struct {
	File *LiteralString

ImportBin represents importbin "file".

type ImportStr

type ImportStr struct {
	File *LiteralString

ImportStr represents importstr "file".

type InSuper

type InSuper struct {
	Index       Node
	InFodder    Fodder
	SuperFodder Fodder

InSuper represents the e in super construct.

type Index

type Index struct {
	Target Node
	Index  Node
	// When Index is being used, this is the fodder before the ']'.
	// When Id is being used, this is the fodder before the id.
	RightBracketFodder Fodder
	// When Index is being used, this is the fodder before the '['.
	// When Id is being used, this is the fodder before the '.'.
	LeftBracketFodder Fodder
	//nolint: golint,stylecheck // keeping Id instead of ID for now to avoid breaking 3rd parties
	Id *Identifier

Index represents both e[e] and the syntax sugar e.f.

One of index and id will be nil before desugaring. After desugaring id will be nil.

type LiteralBoolean

type LiteralBoolean struct {
	Value bool

LiteralBoolean represents true and false

type LiteralNull

type LiteralNull struct{ NodeBase }

LiteralNull represents the null keyword

type LiteralNumber

type LiteralNumber struct {
	OriginalString string

LiteralNumber represents a JSON number

type LiteralString

type LiteralString struct {
	Value           string
	BlockIndent     string
	BlockTermIndent string
	Kind LiteralStringKind

LiteralString represents a JSON string

type LiteralStringKind

type LiteralStringKind int

LiteralStringKind represents the kind of a literal string.

const (
	StringSingle LiteralStringKind = iota

Literal string kinds

func (LiteralStringKind) FullyEscaped

func (k LiteralStringKind) FullyEscaped() bool

FullyEscaped returns true iff the literal string kind may contain escape sequences that require unescaping.

type Local

type Local struct {
	Binds LocalBinds
	Body  Node

Local represents local x = e; e. After desugaring, functionSugar is false.

type LocalBind

type LocalBind struct {
	VarFodder Fodder
	// If Fun is set then its body == Body.
	Body     Node
	EqFodder Fodder
	Variable Identifier
	// The fodder before the closing ',' or ';' (whichever it is)
	CloseFodder Fodder
	// There is no base fodder in Fun because there was no `function` keyword.
	Fun      *Function
	LocRange LocationRange

LocalBind is a helper struct for astLocal

type LocalBinds

type LocalBinds []LocalBind

LocalBinds represents a LocalBind slice.

type Location

type Location struct {
	Line int
	// Column is a byte offset from the beginning of the line
	Column int

Location represents a single location in an (unspecified) file.

func (*Location) IsSet

func (l *Location) IsSet() bool

IsSet returns if this Location has been set.

func (*Location) String

func (l *Location) String() string

type LocationRange

type LocationRange struct {
	File *Source
	// FileName should be the imported path or "" for snippets etc.
	FileName string
	Begin    Location
	End      Location // TODO(sbarzowski) inclusive? exclusive? a gap?

LocationRange represents a range of a source file.

func LineBeginning

func LineBeginning(loc *LocationRange) LocationRange

LineBeginning returns the part of a line directly before LocationRange for example:

local x = foo()
          ^^^^^ <- LocationRange loc
local x = foo()
^^^^^^^^^^ <- lineBeginning(loc)

func LineEnding

func LineEnding(loc *LocationRange) LocationRange

LineEnding returns the part of a line directly after LocationRange for example:

local x = foo() + test
          ^^^^^ <- LocationRange loc
local x = foo() + test
               ^^^^^^^ <- lineEnding(loc)

func LocationRangeBetween

func LocationRangeBetween(a, b *LocationRange) LocationRange

LocationRangeBetween returns a LocationRange containing both a and b.

func MakeLocationRange

func MakeLocationRange(fn string, fc *Source, begin Location, end Location) LocationRange

MakeLocationRange creates a LocationRange.

func MakeLocationRangeMessage

func MakeLocationRangeMessage(msg string) LocationRange

MakeLocationRangeMessage creates a pseudo-LocationRange with a message but no location information. This is useful for special locations, e.g. manifestation entry point.

func (*LocationRange) IsSet

func (lr *LocationRange) IsSet() bool

IsSet returns if this LocationRange has been set.

func (*LocationRange) String

func (lr *LocationRange) String() string

func (*LocationRange) WithCode

func (lr *LocationRange) WithCode() bool

WithCode returns true iff the LocationRange is linked to code. TODO: This is identical to lr.IsSet(). Is it required at all?

type NamedArgument

type NamedArgument struct {
	NameFodder  Fodder
	Name        Identifier
	EqFodder    Fodder
	Arg         Node
	CommaFodder Fodder

NamedArgument represents a named argument to function call x=1.

type Node

type Node interface {
	Context() Context
	Loc() *LocationRange
	FreeVariables() Identifiers
	// OpenFodder returns the fodder before the first token of an AST node.
	// Since every AST node has opening fodder, it is defined here.
	// If the AST node is left recursive (e.g. BinaryOp) then it is ambiguous
	// where the fodder should be stored.  This is resolved by storing it as
	// far inside the tree as possible.  OpenFodder returns a pointer to allow
	// the caller to modify the fodder.
	OpenFodder() *Fodder

Node represents a node in the AST.

func Clone

func Clone(astPtr Node) Node

Clone creates an independent copy of an AST

type NodeBase

type NodeBase struct {
	// This is the fodder that precedes the first token of the node.
	// If the node is left-recursive, i.e. the first token is actually
	// a token of a sub-expression, then Fodder is nil.
	Fodder   Fodder
	Ctx      Context
	FreeVars Identifiers
	LocRange LocationRange

NodeBase holds fields common to all node types.

func NewNodeBase

func NewNodeBase(loc LocationRange, fodder Fodder, freeVariables Identifiers) NodeBase

NewNodeBase creates a new NodeBase from initial LocationRange and Identifiers.

func NewNodeBaseLoc

func NewNodeBaseLoc(loc LocationRange, fodder Fodder) NodeBase

NewNodeBaseLoc creates a new NodeBase from an initial LocationRange.

func (*NodeBase) Context

func (n *NodeBase) Context() Context

Context returns a NodeBase's context.

func (*NodeBase) FreeVariables

func (n *NodeBase) FreeVariables() Identifiers

FreeVariables returns a NodeBase's freeVariables.

func (*NodeBase) Loc

func (n *NodeBase) Loc() *LocationRange

Loc returns a NodeBase's loc.

func (*NodeBase) OpenFodder added in v0.15.0

func (n *NodeBase) OpenFodder() *Fodder

OpenFodder returns a NodeBase's opening fodder.

func (*NodeBase) SetContext

func (n *NodeBase) SetContext(context Context)

SetContext sets a NodeBase's context.

func (*NodeBase) SetFreeVariables

func (n *NodeBase) SetFreeVariables(idents Identifiers)

SetFreeVariables sets a NodeBase's freeVariables.

type Nodes

type Nodes []Node

Nodes represents a Node slice.

type Object

type Object struct {
	Fields      ObjectFields
	CloseFodder Fodder
	TrailingComma bool

Object represents object constructors { f: e ... }.

The trailing comma is only allowed if len(fields) > 0. Converted to DesugaredObject during desugaring.

type ObjectComp

type ObjectComp struct {
	Fields              ObjectFields
	TrailingCommaFodder Fodder
	CloseFodder         Fodder
	Spec                ForSpec
	TrailingComma bool

ObjectComp represents object comprehension

{ [e]: e for x in e for.. if... }.

type ObjectField

type ObjectField struct {
	// f(x, y, z): ...  (ignore if kind  == astObjectAssert)
	// If Method is set then Expr2 == Method.Body.
	// There is no base fodder in Method because there was no `function`
	// keyword.
	Method *Function
	//nolint: golint,stylecheck // keeping Id instead of ID for now to avoid breaking 3rd parties
	Id           *Identifier
	Fodder2      Fodder
	Fodder1      Fodder
	OpFodder     Fodder
	CommaFodder  Fodder
	Expr1        Node // Not in scope of the object
	Expr2, Expr3 Node // In scope of the object (can see self).
	LocRange     LocationRange
	Kind         ObjectFieldKind
	Hide         ObjectFieldHide // (ignore if kind != astObjectFieldID/Expr/Str)
	SuperSugar   bool            // +:  (ignore if kind != astObjectFieldID/Expr/Str)

ObjectField represents a field of an object or object comprehension. TODO(sbarzowski) consider having separate types for various kinds

func ObjectFieldLocalNoMethod

func ObjectFieldLocalNoMethod(id *Identifier, body Node, loc LocationRange) ObjectField

ObjectFieldLocalNoMethod creates a non-method local object field.

type ObjectFieldHide

type ObjectFieldHide int

ObjectFieldHide represents the visibility of an object field.

const (
	ObjectFieldHidden  ObjectFieldHide = iota // f:: e
	ObjectFieldInherit                        // f: e
	ObjectFieldVisible                        // f::: e

Object field visibilities

type ObjectFieldKind

type ObjectFieldKind int

ObjectFieldKind represents the kind of an object field.

const (

	// <f1> 'assert' <expr2> '[' <opF> ':' <expr3> ']' <commaF>
	// where expr3 can be nil
	ObjectAssert ObjectFieldKind = iota
	// <f1> <id> <colon> <expr2> <commaF>
	// <f1> '[' <expr1> <f2> ']' <colon> <expr2> <commaF>
	// <expr1> <colon> <expr2> <commaF>
	// <f1> 'local' <f2> <id> '=' <expr2> <commaF>

Kinds of object fields

type ObjectFields

type ObjectFields []ObjectField

ObjectFields represents an ObjectField slice.

type Parameter added in v0.16.0

type Parameter struct {
	NameFodder  Fodder
	Name        Identifier
	CommaFodder Fodder
	EqFodder    Fodder
	DefaultArg  Node
	LocRange    LocationRange

Parameter represents a parameter of function. If DefaultArg is set, it's an optional named parameter. Otherwise, it's a positional parameter and EqFodder is not used.

type Parens

type Parens struct {
	Inner       Node
	CloseFodder Fodder

Parens represents parentheses

( e )

type Self

type Self struct{ NodeBase }

Self represents the self keyword.

type Slice

type Slice struct {
	Target            Node
	LeftBracketFodder Fodder
	// Each of these can be nil
	BeginIndex         Node
	EndColonFodder     Fodder
	EndIndex           Node
	StepColonFodder    Fodder
	Step               Node
	RightBracketFodder Fodder

Slice represents an array slice a[begin:end:step].

type Source

type Source struct {
	// DiagnosticFileName is the imported path or a special string
	// for indicating stdin, extvars and other non-imported sources.
	DiagnosticFileName DiagnosticFileName
	Lines              []string

Source represents a source file.

func BuildSource

func BuildSource(dFilename DiagnosticFileName, s string) *Source

BuildSource transforms a source file string into a Source struct. TODO: This seems like a job for strings.Split() with a final \n touch-up.

type SourceProvider

type SourceProvider struct {

SourceProvider represents a source provider. TODO: Need an explanation of why this exists.

func (*SourceProvider) GetSnippet

func (sp *SourceProvider) GetSnippet(loc LocationRange) string

GetSnippet returns a code snippet corresponding to loc.

type SuperIndex

type SuperIndex struct {
	// If super.f, the fodder before the 'f'
	// If super[e], the fodder before the ']'.
	IDFodder Fodder
	Index    Node
	// If super.f, the fodder before the '.'
	// If super[e], the fodder before the '['.
	DotFodder Fodder
	//nolint: golint,stylecheck // keeping Id instead of ID for now to avoid breaking 3rd parties
	Id *Identifier

SuperIndex represents the super[e] and super.f constructs.

Either index or identifier will be set before desugaring. After desugaring, id will be nil.

type Unary

type Unary struct {
	Expr Node
	Op UnaryOp

Unary represents unary operators.

type UnaryOp

type UnaryOp int

UnaryOp represents a unary operator.

const (
	UopNot UnaryOp = iota

Unary operators

func (UnaryOp) String

func (u UnaryOp) String() string

type Var

type Var struct {
	//nolint: golint,stylecheck // keeping Id instead of ID for now to avoid breaking 3rd parties
	Id Identifier

Var represents variables.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL