ast

Documentation

Overview

Package ast provides a basic tree implementation for storing an AST generated by a parser.

The AST is represented as a []byte. The first four bytes contain the format version, the type of the AST, and the main language of the file. The last byte contains the offset to the root node (there is always exactly one root node).

Siblings are adjacent in the buffer and children always preceed parents. For declaratives ASTs, interesting substrings of the source text (e.g., identifier names) are placed before the first set of siblings in which they are referenced, and they are reused for later occurences of the same text.

Current layout of a serialized node:

• Flags (uint8, required)
• Relative offset to previous sibling (uint8, required)
• Type (uint16, required)
• Relative offset to parent (uint32, required)
• offset within source file (uint16 or uint32, optional)
• (Full-only) endOffset within source file (uint16 or uint32, optional) OR (Declarative-only) Relative offset to the node's text (uint16 or uint32, optional)
• Relative offset to first child (uint16 or uint32, optional)

Index

• func AlwaysRecover(err parsers.SyntaxError) bool
• type Node
  • func (n Node) Child(sel node.Selector) Node
  • func (n Node) Children(sel node.Selector) []Node
  • func (n Node) ChildrenOfType(t node.Type) (nodes []Node)
  • func (n Node) Container(sel node.Selector) Node
  • func (n Node) Descend(sels ...node.Selector) Node
  • func (n Node) EndOffset() int
  • func (n Node) FirstChild() Node
  • func (n Node) FirstChildOfType(t node.Type) Node
  • func (n Node) Index(sel node.Selector) int
  • func (n Node) IsValid() bool
  • func (n Node) LastChildOfType(t node.Type) Node
  • func (n Node) LocalID() uint32
  • func (n Node) Next(sel node.Selector) Node
  • func (n Node) NextAll(sel node.Selector) []Node
  • func (n Node) NextSibling() Node
  • func (n Node) Offset() int
  • func (n Node) Parent() Node
  • func (n Node) Prev(sel node.Selector) Node
  • func (n Node) PrevSibling() Node
  • func (n Node) Range() offset.Range
  • func (n Node) String() string
  • func (n Node) Text() string
  • func (n Node) Traverse(visitor func(n Node) bool)
  • func (n Node) Tree() *Tree
  • func (n Node) Type() node.Type
• type Options
• type Tree
  • func Build(ctx context.Context, l lpb.Language, path, source string, opts Options) (*Tree, error)
  • func FromBytes(bytes []byte, path, source string) (*Tree, error)
  • func (tree *Tree) Buffer() []byte
  • func (tree *Tree) ForEach(sel node.Selector, consumer func(n Node))
  • func (tree *Tree) Language() lpb.Language
  • func (tree *Tree) Mapper() *offset.Mapper
  • func (tree *Tree) NodeByID(localID uint32) Node
  • func (tree *Tree) NodeByOffset(offset int) Node
  • func (tree *Tree) NodeByRange(r offset.Range) Node
  • func (tree *Tree) Path() string
  • func (tree *Tree) Root() Node
  • func (tree *Tree) Text() string
  • func (tree *Tree) Type() Type
• type Type

Functions

func AlwaysRecover

func AlwaysRecover(err parsers.SyntaxError) bool

AlwaysRecover turns on unconditional recovery if found in the Options.ShouldTryToRecover field.

Types

type Node

type Node struct {
	// contains filtered or unexported fields
}

Node is an element of the AST. It stores information necessary to extract the node data (type, offsets) and the relatives of the node (parent, siblings, etc.) from the AST buffer.

A Node is intended to be passed by value (currently 16 bytes).

Node follows the Null Object Pattern. Getting a non-existent relative of a Node returns an empty Node object, for which IsValid() returns false and all other methods return default values.

func (Node) Child

func (n Node) Child(sel node.Selector) Node

Child returns the first child that matches the provided selector.

func (Node) Children

func (n Node) Children(sel node.Selector) []Node

Children returns all children nodes that match the provided selector.

func (Node) ChildrenOfType

func (n Node) ChildrenOfType(t node.Type) (nodes []Node)

ChildrenOfType returns all the direct children with the specified type.

func (Node) Container

func (n Node) Container(sel node.Selector) Node

Container returns the innermost container node that matches the provided selector.

func (Node) Descend

func (n Node) Descend(sels ...node.Selector) Node

Descend descends into the AST layer by layer picking the next node among children of the previous node. This method is equivalent to a chain of Child() calls.

func (Node) EndOffset

func (n Node) EndOffset() int

EndOffset returns the node's 0-based byte end offset in the source content.

func (Node) FirstChild

func (n Node) FirstChild() Node

FirstChild returns the first child of the node or an invalid node if it there isn't any.

func (Node) FirstChildOfType

func (n Node) FirstChildOfType(t node.Type) Node

FirstChildOfType returns the first child with the specified type or an invalid node if not found.

func (Node) Index

func (n Node) Index(sel node.Selector) int

Index returns the 0-based position of the node among its siblings matching the given selector. Returns -1 for invalid nodes.

func (Node) IsValid

func (n Node) IsValid() bool

IsValid returns whether the node exists. For example, n.NextSibling().IsValid() returns false if n has no next sibling.

func (Node) LastChildOfType

func (n Node) LastChildOfType(t node.Type) Node

LastChildOfType returns the last child with the specified type or an invalid node if not found.

func (Node) LocalID

func (n Node) LocalID() uint32

LocalID returns a unique id of this node within the current file.

func (Node) Next

func (n Node) Next(sel node.Selector) Node

Next returns the first element among the following siblings that matches the selector.

func (Node) NextAll

func (n Node) NextAll(sel node.Selector) []Node

NextAll returns all following siblings of the node that match the selector.

func (Node) NextSibling

func (n Node) NextSibling() Node

NextSibling returns the next sibling of a node or an invalid node if there isn't any.

func (Node) Offset

func (n Node) Offset() int

Offset returns the node's 0-based byte offset in the source content.

func (Node) Parent

func (n Node) Parent() Node

Parent returns the node's parent node or an invalid node if there isn't any.

func (Node) Prev

func (n Node) Prev(sel node.Selector) Node

Prev returns the first element among the previous siblings that matches the selector.

func (Node) PrevSibling

func (n Node) PrevSibling() Node

PrevSibling returns the previous sibling of a node or an invalid node if there isn't any.

func (Node) Range

func (n Node) Range() offset.Range

Range returns the range of the node in the source content. This is more efficient than calling n.Offset() and n.EndOffset() separately.

func (Node) String

func (n Node) String() string

func (Node) Text

func (n Node) Text() string

Text returns a substring of the source text, bounded by the offsets: [offset, endOffset). If the offsets are invalid, an empty string is returned. For declarative ASTs, Text() only returns non-empty for nodes whose text has been nested in the serialized AST (e.g., identifiers).

func (Node) Traverse

func (n Node) Traverse(visitor func(n Node) bool)

Traverse does a pre-order traversal of the subtree starting at this node. The provided visitor method can return false to skip children of the visited node.

func (Node) Tree

func (n Node) Tree() *Tree

Tree returns the tree that the node belongs to.

func (Node) Type

func (n Node) Type() node.Type

Type returns the node's type.

type Options

type Options struct {
	// Adds node.Punctuation and node.Keyword nodes to the parse tree.
	IncludeAllTokens bool

	// A callback function which decides whether the parser should try to recover
	// and continue parsing. If it decides to abort the processing, the last error
	// gets returned as the main outcome from the parser. Successful error
	// recovery leads to one or more SyntaxProblem or InvalidToken nodes in
	// the tree.
	//
	// Never called on syntactically valid input.
	// Leave unset to disable error recovery.
	ShouldTryToRecover func(err parsers.SyntaxError) bool

	// Expand the regions of some nodes to include preceding comments.
	AttachCommentsToNodes bool

	// The type of AST to construct.
	Type Type
}

Options contains parameters that control the shape of the produced AST.

type Tree

type Tree struct {
	// contains filtered or unexported fields
}

Tree represents a language independent tree representation of the code as produced by parsers.

func Build

func Build(ctx context.Context, l lpb.Language, path, source string, opts Options) (*Tree, error)

Build produces an AST for the input source, built by the parser registered for the language. Returns a half-constructed tree with BrokenFile as the root, and an error if "source" cannot be parsed.

func FromBytes

func FromBytes(bytes []byte, path, source string) (*Tree, error)

FromBytes returns a Tree backed by the provided bytes, or an error if it couldn't be created. For full ASTs, if the source is not provided, Text() will return an empty string for all nodes.

func (*Tree) Buffer

func (tree *Tree) Buffer() []byte

Buffer returns a slice of bytes that contain the serialized version of the AST, to be used for transmission and storage.

func (*Tree) ForEach

func (tree *Tree) ForEach(sel node.Selector, consumer func(n Node))

ForEach iterates over all AST nodes that match a given selector (children before parents).

func (*Tree) Language

func (tree *Tree) Language() lpb.Language

Language returns the language of the file.

func (*Tree) Mapper

func (tree *Tree) Mapper() *offset.Mapper

Mapper returns the tree's offset mapper for exposing offset helper methods.

func (*Tree) NodeByID

func (tree *Tree) NodeByID(localID uint32) Node

NodeByID returns a node by its local ID.

func (*Tree) NodeByOffset

func (tree *Tree) NodeByOffset(offset int) Node

NodeByOffset returns the most specific node found at that offset or an invalid node if that offset is not covered by any node. If the cursor is placed between two touching nodes, we prefer the one that looks like an identifier (an identifier cannot start right after another identifier) and by default take the first one.

func (*Tree) NodeByRange

func (tree *Tree) NodeByRange(r offset.Range) Node

NodeByRange returns the innermost node that fully contains the given range. Empty ranges are not supported.

func (*Tree) Path

func (tree *Tree) Path() string

Path returns the location of the parsed source code.

Note: this function returns what was provided during the AST creation, so this can be either a relative or absolute path, or even a URI, depending on where this tree is coming from.

func (*Tree) Root

func (tree *Tree) Root() Node

Root returns the tree's root node.

func (*Tree) Text

func (tree *Tree) Text() string

Text returns the underlying source code (for full ASTs only)

func (*Tree) Type

func (tree *Tree) Type() Type

Type returns the type of the tree.

type Type

type Type uint8

Type is the type of the AST.

const (
	// Full ASTs contain are intended to be used for local modifications and refactorings of the
	// currently open file. They contain all node types and can reference the file content, provided
	// as a separate string.
	Full Type = 0
	// Declarative ASTs are intended to be cached for "background" files (all files except for the one
	// currently being edited). They don’t have comments, keywords, punctuation characters and method
	// bodies and don't support the Offset() and EndOffset() methods. However, relevant parts of the
	// source text are stored directly within the serialized buffer (e.g., identifier names).
	Declarative Type = 1
)