collector

README

pkg/analyzer/collector — CST → AST

Converts a tree-sitter CST into *ast.Program and *symbols.SymbolTable.

Entry point: collector.NewCollector(source []byte) → c.Collect(rootNode) (program, table, errors)

Dispatch: collector.go owns CollectStatement and CollectExpr (switch on node.Kind()), and ParseType. Subpackages call back into the root collector via the Collector interface to avoid circular imports.

Canonical-type resolution (canonical.go): Collect finishes with resolveCanonicalTypes, which stamps TypeDeclStmt.CanonicalKind ("Result"/"Maybe"/"") — the single source of truth for whether a type is the compiler-known Result/Maybe that ?, must-use, ??, and the try-context check key off. Identity is conferred by a @builtin(Result)/@builtin(Maybe) attribute (collected onto TypeDeclStmt.Builtin via collectBuiltin, reusing the @derive attribute grammar — no grammar change), which is name-independent (a type named Either can be the canonical Result) but shape-validated; with no marker, an unmarked type literally named "Result"/"Maybe" with the canonical constructor shape is stamped as a fallback — the path a program with no prelude in its search roots takes, which is most tests, though std/prelude.lyra now marks its own types so a normal build goes through the marker. A malformed marker (wrong shape, unknown kind, duplicate claim) is lyra-E017. Recognition sites read the stamp via the symbol table and keep a name+arity fallback only for a truly undeclared ambient annotation.

Once a marker claims a kind, a same-named unmarked declaration is an ordinary type — right, but it used to surface as `?` operand must be a Result or Maybe, got Maybe. So the same pass also stamps ShadowedCanonical (the kind the declaration looks like but is not) and ShapeMatchesCanonical (whether it would otherwise have qualified), which is all ? needs to say whether the author re-declared the prelude's type or gave an unrelated type its name. It is stamped here rather than re-derived at the diagnostic, so the shape test has one home.

Two traps that pass live in that stamp. It walks the statement list rather than reading c.table.Types[kind], because a declaration shadowing a prelude name is keyed <module>::<name> so the prelude keeps the bare key — the lookup returns the prelude's declaration, the one this is not about. And the advice it enables must never be "mark it @builtin(Maybe) too": that is a duplicate claim, lyra-E017, so the message says remove or rename instead.

Struct-pattern reclassification (reclassifyStructPatterns, collector.go): after walkProgram, Collect walks every pattern site (match arms, destructuring lets, if let/else, lambda params/clauses — via ast.WalkStmt/WalkExpr to reach the containers) and rewrites a DataPattern whose name is a declared struct type into a named StructPattern (reclassifyPattern, recursing into sub-patterns). Needed because Pt { x, y } (a struct pattern) and Node { l, r } (a data-constructor inline-record pattern) are syntactically identical — both parse to a DataPattern with an inner StructPattern payload — so the split is semantic and can only run once the symbol table is complete (a forward-referenced struct still resolves). A name that is a data constructor (or unknown) stays a DataPattern. Downstream (typechecker, backend) therefore sees StructPattern for structs and DataPattern for data variants, no per-site "is this name a struct?" branching.

Constructor-expression reclassification (reclassifyConstructorExprs, constructor_reclassify.go): the same post-pass idea for expression position. An all-caps / single-capital constructor or named-tuple name (data Dir = N | S | E | W, tuple POINT(…)) lexes as a const_identifier (the token reserved for constants, /[A-Z][A-Z0-9_]*/; user_defined_type_name needs a lowercase letter to be unambiguous), so a bare use collects to an IdentifierExpr and an applied use FOO(3) to a FunctionCallExpr — not the DataConstructorExpr / named TupleLiteralExpr a PascalCase constructor yields, and the typechecker then reported a misleading "undefined identifier". This pass (run after the symbol table is complete, so a forward-referenced constructor resolves) rewrites a bare nullary-constructor name into a DataConstructorExpr and an applied constructor/named-tuple call into a named TupleLiteralExpr — the exact nodes PascalCase produces — so all downstream passes handle them identically with no special-casing. It reassigns each expression slot in place through ast.RewriteStmt, the writing half of the canonical walker — it used to mirror ast.walkExprChildren by hand, which a visitor cannot do, and the copy had fallen three node kinds behind (see COMPLETED.md, 08/23). A value binding of the same name (a const N, checked against the global scope) shadows the constructor and skips the rewrite, so existing constant code is untouched. Pattern position already resolved these constructors, so only expressions needed it.

Nil-node hazard: cst.Field(node, ...) returns a genuine Go nil *sitter.Node for an absent optional grammar field (e.g. a zero-parameter lambda_type's parameter_types). Calling any accessor (ChildCount, Child, Kind, …) on that nil node hangs inside the go-tree-sitter CGO binding instead of panicking — found via a real bug (parseParameterTypes, fixed 06/24/26) where this silently froze the whole collector. Always nil-check before touching the result of an optional field lookup, the same way parseType/CollectExpression already do.

Never return a nil expression node into the AST: an expression collector that hits an unrecoverable value error (e.g. a numeric literal that overflows int64) must emit a diagnostic and return a placeholder node (a zero-valued IntegerLiteralExpr/FloatLiteralExpr), never nil. A nil returned up as an ast.Expression becomes a typed-nil interface ((*T)(nil), non-nil interface with a nil pointer) that slips past expr == nil checks and crashes a later pass on the first field access — this is exactly how an out-of-range literal panicked propagateLiteralType (fixed 07/24/26, numeric_literals.go). The error diagnostic keeps the program from compiling, so the placeholder value is inert. The statement analogue (block bodies): CollectBlockExpr skips a child that collects to nil (isNilStmt — untyped and typed nils) rather than appending it, because a block's value is its final statement — a trailing comment (a named CST child that collects to nil) would otherwise become the block's value and miscompile (the backend returned garbage for a + b // c; fixed 07/25/26). A comment-only body collects to an empty block.

Subpackages (all pass *collector_ctx.Ctx as their first argument):

Subpackage	Files handle
declarations/	let/var/const decls, destructuring decls, if let, trait decls, trait impls, module decls
typedecls/	struct, data, named tuples, newtype, constrained types, attributes
expressions/	all expression kinds (one file per kind)
statements/	for, for-in, return, break, continue, with, var reassignment, deref assignment

collector_ctx.Ctx — shared state passed to every subpackage function:

• ctx.Source []byte — raw source bytes
• ctx.NodeText(node) — extract text from a node
• ctx.NodeLocation(node) — convert to 1-based ast.Location
• ctx.AddError(node, severity, format, args...) — append a CollectorError
• ctx.MustField(node, fieldName) — get a required child field, emitting an error if missing
• ctx.Collector — embedded interface for recursive dispatch back to the root

Tree-sitter traversal conventions:

• node.ChildCount() / node.Child(i) — all children including anonymous keyword tokens; use with switch child.Kind()
• cst.Field(node, "field") — first child with that field name. Use this, not node.ChildByFieldName: the two answer identically, but ChildByFieldName allocates a C string from the Go name, calls into C and frees it on every lookup, which made it about a quarter of all samples in an analysis run — the collector asks at nearly every node. cst.Field resolves the name to a grammar field id once and reuses it, and moving the collector onto it made the whole pipeline ~25% faster (pkg/cst, and the benchmarks in pkg/driver/bench_test.go)
• node.FieldNameForChild(uint32(i)) — field name at index i; use when a rule repeats the same field name (e.g. multiple value: fields in commaSep1)

Spellings the collector erases

Two surface forms build the same AST as an existing one, so nothing after the collector learns they exist:

• Juxtaposed constructor application. Some 42 builds the same named TupleLiteralExpr that Some(42) builds (collectAppliedConstructorExpr), so the typechecker, purity, exhaustiveness and the backend never see juxtaposition.
• A bare jump as a match arm body (08/06). None => break builds the single-statement BlockExpr that None => { break } builds (collectMatchArmBody). The jump forms are statements and an arm body is an expression, so the bare spelling did not parse at all before; the braced one already worked end to end.

The second is worth stating as a rule rather than a trick, because the alternative is expensive and looks reasonable: letting MatchArm.Body hold a statement would push the distinction into the typechecker, the purity and ownership passes, and all four of the backend's arm-body lowering sites — every one of which would need a case doing exactly what the block case already does. Erasing at the boundary costs one function.

The invariant to keep: the two spellings must stay byte-identical in the AST. If they ever diverge, the erased form has acquired a meaning of its own and every downstream pass becomes a place the two can differ. TestMatchArm_BareJumpCollectsAsTheBracedForm pins it by collecting both and comparing the printed trees.

Ranges: one notation, one strictness rule

The .. notation has three sites — an expression (0..<n), a match pattern (0..<=9), and a newtype range constraint (range(0..<=100)). Since 08/01 they share one grammar shape (rangeBounds in tree-sitter-lyra/include/helpers.js) and one collector check here:

• ctx.RangeEndOperator(node, form) enforces that a range with an end bound says whether that bound is included (lyra-E032), and returns the operator. The operator is optional in the grammar at all three sites and required here, deliberately: every reader of the collected operator tests == "<", so an omitted one fell through to inclusive and 0..9 silently meant 0..<=9. A diagnostic naming both fixes beats a syntax error pointing at whichever token failed to shift — the same trade lyra-E029 made for modifier order. The suggestion is spliced from the source at the first .., so it is right for open-start (..9) and stepped (0..10:2) forms too. Returns "=" after reporting so the caller still builds a well-formed node (hazard 3).

• collector_ctx.RangeBound(node) answers "was this bound actually written?" — and it is not a nil check. Where the grammar requires a bound, tree-sitter's error recovery can insert one to keep parsing: range(..) yields a zero-width decimal_int sitting on the ). A plain nil check reads that insertion as a bound of value zero, which is how the long-standing "range constraint must have a start or end" check would have started passing silently. Both the missing flag and the empty span are tested, because the recovery does not always set the former.

A nil Start or End on a collected RangePattern therefore means an open range (10.., ..<0), never a malformed one — a bare .. does not parse. Every consumer must read it that way; the ones that exist are the backend's match lowering, the exhaustiveness check's armIntInterval, and the range analysis's pattern refinement, and all three treat an absent bound as the scrutinee type's own limit.

Documentation

Index

• Constants
• type Collector
  • func NewCollector(source []byte) *Collector
  • func (c *Collector) AddFile(root *sitter.Node, source []byte, file, modulePath string)
  • func (c *Collector) Collect(root *sitter.Node) (*ast.Program, *symbols.SymbolTable, *symbols.ScopeTable, []error)
  • func (c *Collector) CollectBounds(node *sitter.Node) []string
  • func (c *Collector) CollectExpr(node *sitter.Node) ast.Expression
  • func (c *Collector) CollectGenericParams(node *sitter.Node) []ast.GenericParam
  • func (c *Collector) CollectPattern(patternNode *sitter.Node) ast.Pattern
  • func (c *Collector) CollectStatement(node *sitter.Node) ast.Statement
  • func (c *Collector) DeclareOverload(stmt *ast.VarDeclStmt) (bool, string)
  • func (c *Collector) DefinePatternBinding(named ast.Named)
  • func (c *Collector) Finish() (*ast.Program, *symbols.SymbolTable, *symbols.ScopeTable, []error)
  • func (c *Collector) LookupCurrentScope(name string) (ast.Named, bool)
  • func (c *Collector) MergeWhereConstraints(params []ast.GenericParam, whereNode *sitter.Node) []ast.GenericParam
  • func (c *Collector) ParseDestructuringPattern(patternNode *sitter.Node) ast.Pattern
  • func (c *Collector) ParseLambdaType(node *sitter.Node) *types.LambdaType
  • func (c *Collector) ParseType(node *sitter.Node) types.Type
  • func (c *Collector) PopScope()
  • func (c *Collector) PushBlockScope() *symbols.Scope
  • func (c *Collector) PushFunctionScope() *symbols.Scope
  • func (c *Collector) PushLoopScope() *symbols.Scope
  • func (c *Collector) RecordScope(node ast.AstNode, scope *symbols.Scope)
  • func (c *Collector) RecordTypeRef(name string, loc ast.Location)
  • func (c *Collector) RedefineVariable(stmt *ast.VarDeclStmt)
  • func (c *Collector) RegisterDestructuredName(name string, decl *ast.DestructuringDeclStmt)
  • func (c *Collector) RegisterFunction(name string, stmt *ast.LambdaExpr) error
  • func (c *Collector) RegisterParameter(p *ast.Parameter) error
  • func (c *Collector) RegisterTrait(stmt *ast.TraitDeclStmt) error
  • func (c *Collector) RegisterType(stmt *ast.TypeDeclStmt) error
  • func (c *Collector) RegisterVariable(stmt *ast.VarDeclStmt) error
  • func (c *Collector) SetImports(graph map[string][]string)
  • func (c *Collector) SetPreludeModule(path string)
  • func (c *Collector) Snapshot() *Snapshot
• type Snapshot
  • func (s *Snapshot) Restore(source []byte) *Collector
  • func (s *Snapshot) StatementCount() int

Constants

const (
	CollectorErrorSeverityError   = diag.SeverityError
	CollectorErrorSeverityWarning = diag.SeverityWarning
	CollectorErrorSeverityInfo    = diag.SeverityInfo
)

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector walks the CST and builds an AST + symbol table.

func NewCollector

func NewCollector(source []byte) *Collector

func (*Collector) AddFile

func (c *Collector) AddFile(root *sitter.Node, source []byte, file, modulePath string)

AddFile walks one source file into the accumulating program, and may be called repeatedly — once per module — before Finish.

The split exists because only the *walk* is per-file: every pass in Finish reaches across the whole program. reclassifyStructPatterns has to know every declared struct, resolveCanonicalTypes every declared type, and registerTopLevelFunctions every top-level binding — and with modules those can come from a file that has not been walked yet. Running them per file would make the result depend on collection order.

source is swapped in alongside the tree because every text and location read goes through it (ctx.NodeText / ctx.NodeLocation); a stale source would slice the wrong bytes for the file being walked.

func (*Collector) Collect

func (c *Collector) Collect(root *sitter.Node) (*ast.Program, *symbols.SymbolTable, *symbols.ScopeTable, []error)

Collect walks the entire tree and returns the AST, symbol table, scope table, and any errors. It is AddFile + Finish for the single-file case.

func (*Collector) CollectBounds

func (c *Collector) CollectBounds(node *sitter.Node) []string

CollectBounds reads the trait names in a bound list — a `where` clause's, an inline `<t: Shown>`, a trait's supertraits, an impl's constraints. It is the funnel for all four, which is why recording their spans here covers every one.

A bound is a `[]string` rather than a type, so these names never pass through `parseType` and the span it records; without this, a cursor on `Shown` in `where t: Shown` resolved to nothing while one on `Point` in `(p: Point)` resolved.

func (*Collector) CollectExpr

func (c *Collector) CollectExpr(node *sitter.Node) ast.Expression

func (*Collector) CollectGenericParams

func (c *Collector) CollectGenericParams(node *sitter.Node) []ast.GenericParam

func (*Collector) CollectPattern

func (c *Collector) CollectPattern(patternNode *sitter.Node) ast.Pattern

func (*Collector) CollectStatement

func (c *Collector) CollectStatement(node *sitter.Node) ast.Statement

func (*Collector) DeclareOverload

func (c *Collector) DeclareOverload(stmt *ast.VarDeclStmt) (bool, string)

DeclareOverload offers stmt to the same-named declaration already in scope as a receiver-keyed overload.

**Only at module level.** Overloading is a property of a module's public surface, and a set is resolved by looking one up in a module scope; a `let` inside a function body that happens to reuse a name is sequential rebinding, which is a different feature with a different meaning, and admitting overloads there would change what the second `let` means based on whether the first took a `self` parameter.

func (*Collector) DefinePatternBinding

func (c *Collector) DefinePatternBinding(named ast.Named)

DefinePatternBinding enters a pattern's binding into the current scope.

Errors are deliberately dropped. `Define` reports a redeclaration, which is the right answer for two `let`s of one name in a block and the wrong one here: a pattern may bind a name that shadows an outer one, which is ordinary and intended, and an arm is its own scope so the only way to collide *within* it is a malformed pattern the pattern collector has already reported.

func (*Collector) Finish

func (c *Collector) Finish() (*ast.Program, *symbols.SymbolTable, *symbols.ScopeTable, []error)

Finish runs the whole-program passes and returns the merged result.

func (*Collector) LookupCurrentScope

func (c *Collector) LookupCurrentScope(name string) (ast.Named, bool)

LookupCurrentScope returns the symbol registered under name in the current (innermost) scope only, without walking parent scopes.

func (*Collector) MergeWhereConstraints

func (c *Collector) MergeWhereConstraints(params []ast.GenericParam, whereNode *sitter.Node) []ast.GenericParam

func (*Collector) ParseDestructuringPattern

func (c *Collector) ParseDestructuringPattern(patternNode *sitter.Node) ast.Pattern

func (*Collector) ParseLambdaType

func (c *Collector) ParseLambdaType(node *sitter.Node) *types.LambdaType

func (*Collector) ParseType

func (c *Collector) ParseType(node *sitter.Node) types.Type

func (*Collector) PopScope

func (c *Collector) PopScope()

func (*Collector) PushBlockScope

func (c *Collector) PushBlockScope() *symbols.Scope

func (*Collector) PushFunctionScope

func (c *Collector) PushFunctionScope() *symbols.Scope

func (*Collector) PushLoopScope

func (c *Collector) PushLoopScope() *symbols.Scope

func (*Collector) RecordScope

func (c *Collector) RecordScope(node ast.AstNode, scope *symbols.Scope)

func (*Collector) RecordTypeRef

func (c *Collector) RecordTypeRef(name string, loc ast.Location)

func (*Collector) RedefineVariable

func (c *Collector) RedefineVariable(stmt *ast.VarDeclStmt)

RedefineVariable replaces an existing same-scope binding with stmt, used for same-scope sequential rebinding so that later references resolve to the newest declaration.

func (*Collector) RegisterDestructuredName

func (c *Collector) RegisterDestructuredName(name string, decl *ast.DestructuringDeclStmt)

func (*Collector) RegisterFunction

func (c *Collector) RegisterFunction(name string, stmt *ast.LambdaExpr) error

func (*Collector) RegisterParameter

func (c *Collector) RegisterParameter(p *ast.Parameter) error

func (*Collector) RegisterTrait

func (c *Collector) RegisterTrait(stmt *ast.TraitDeclStmt) error

func (*Collector) RegisterType

func (c *Collector) RegisterType(stmt *ast.TypeDeclStmt) error

func (*Collector) RegisterVariable

func (c *Collector) RegisterVariable(stmt *ast.VarDeclStmt) error

func (*Collector) SetImports

func (c *Collector) SetImports(graph map[string][]string)

SetImports hands over the whole import graph before the first file is walked, for the same reason SetPreludeModule is called there: a declaration taking a name an imported module exports is keyed apart from it (symbols.declKeyIn), and a type is registered under that key *during* the walk.

Assembling the graph file by file as each is walked would work for a single-file module and quietly fail for the rest: a module whose `import` sits in its second file would key the first file's types as though nothing were imported, and the key a lookup computes afterwards — with the graph complete — would miss them. The graph is known before collection anyway, since resolving it is what produced the units.

func (*Collector) SetPreludeModule

func (c *Collector) SetPreludeModule(path string)

SetPreludeModule names the implicitly-imported module, so registration can tell a user declaration taking a prelude name (allowed, warned) from two user modules clashing (an error).

func (*Collector) Snapshot

func (c *Collector) Snapshot() *Snapshot

Snapshot captures the collector's current state. The receiver may go on being used — the capture is a copy, so what happens to it afterwards does not reach the snapshot.

type Snapshot

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is the collector's state after some prefix of a program's files, held so the rest can be collected onto a copy of it rather than onto a re-collection of everything.

**The prefix is what does not change between keystrokes.** A language server re-analyzes a document's whole import graph on every edit; for a small file with the standard prelude that is 12 units of which 11 cannot have changed, and collection is 75% of the analysis.

A snapshot is **immutable once taken**: Restore hands out a deep copy and the original is never written to again. That is the difference between this and undoing the edited file's contributions — a copy has to be right once, an undo has to be right every time, and the failure mode of getting an undo slightly wrong is analysis that drifts as a session runs.

func (*Snapshot) Restore

func (s *Snapshot) Restore(source []byte) *Collector

Restore builds a collector holding a copy of the snapshot's state, ready for the remaining files to be added to it.

The AST statements are shared rather than copied, which is the point: every side table downstream — ScopeTable, TypeTable, MethodTable — is keyed by AST *pointer*, so copying the nodes would invalidate all of them. What makes that safe is that re-running the analysis passes over one collected AST is idempotent, which pkg/driver's re-analysis test checks directly over the real prelude rather than assuming.

func (*Snapshot) StatementCount

func (s *Snapshot) StatementCount() int

StatementCount is how many top-level statements the snapshot covers — the boundary the typechecker resumes from. Read from the snapshot rather than recomputed from the units, because Finish may append a statement (a synthesized derive) and the count at capture time is the only thing that says where the prefix actually ends.