ast

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 31, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package ast models a structural symbol graph that lives alongside the rag chunk store. Where rag's chunks carry only text + position, an Extractor produces a SymbolGraph of named program entities (functions, methods, types, top-level bindings) with edges (call relationships) between them. Together a chunk store and a symbol graph give the retriever both lexical recall and structural reach.

Scope and drift signature

Every SymbolGraph is stamped with both Scope and VectorSpaceID. Scope is the caller-defined corpus/root/index namespace; VectorSpaceID is the opaque embedding-space identity used by the parallel chunk store. Stores key graph rows by (scope, vectorSpaceID), never by vectorSpaceID alone, so two corpora indexed with the same embedder cannot delete or read each other's graph.

When the chunk store re-embeds under a new vector space, the symbol graph must be re-extracted and re-stamped. Nodes and edges from the old vsid become unreachable within the same scope in one atomic step. This invariant exists at the graph level only; individual nodes do not carry their own scope/vsid.

Storage shape

SymbolGraph is a write-only transport structure: an extractor produces one, a SymbolStore consumes one. Consumers MUST NOT traverse the graph by linear scan of its slices — the store builds the indexes needed for O(1) lookup and exposes them via its read methods. The graph itself is permitted to be unindexed and unsorted.

Trust boundary

SymbolNode.Declaration and SymbolNode.Doc carry raw source content from arbitrary files an extractor was pointed at. Extractors MUST NOT pre-sanitize them — the data is structural truth. Any consumer that surfaces these fields to an LLM (MCP tool outputs, chat prompts) owns the sanitization required to mitigate prompt injection from hostile source comments. This package guarantees fidelity, not safety.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrSymbolNotFound indicates GetSymbol or SymbolEnclosing could not
	// resolve the requested symbol or location to a node in the store.
	ErrSymbolNotFound = errors.New("rag/ast: symbol not found")

	// ErrVectorSpaceMismatch indicates a read attempted to use a vsid not
	// present in the store. Distinct from rag.ErrVectorSpaceMismatch
	// because the symbol graph may drift independently of the chunk store
	// during partial migrations.
	ErrVectorSpaceMismatch = errors.New("rag/ast: vector-space mismatch")

	// ErrInvalidGraph indicates a graph or edge violates the SymbolStore
	// persistence contract.
	ErrInvalidGraph = errors.New("rag/ast: invalid graph")

	// ErrInvalidArgument indicates a caller supplied an invalid read argument
	// such as a negative limit.
	ErrInvalidArgument = errors.New("rag/ast: invalid argument")
)

Sentinel errors. Distinct from rag's own sentinels because the recovery action diverges: rag/ vsid mismatches mean "re-embed"; rag/ast vsid mismatches mean "re-extract the AST graph". Both can happen independently when callers mix-and-match stores.

Error strings are namespaced "rag/ast: ..." to match the full import path and disambiguate from go/ast in log aggregators.

Functions

func SymbolID 

func SymbolID(key SymbolKey) string

SymbolID derives the stable identity used as the endpoint of every edge. It is language-neutral: each key component is base64url encoded before joining, so names containing punctuation, separators, or non-Go syntax do not collide. Extractors are responsible for filling Disambiguator whenever their language can have multiple distinct symbols with the same language, kind, namespace, receiver, and name.

Types

type CallEdge 

type CallEdge struct {
	CallerID   string
	CalleeID   string         // resolved SymbolID; required only when resolved
	CalleeRaw  string         // identifier/expression as it appeared in source
	Resolution CallResolution // explicit resolution state
	File       string         // canonicalized per [SymbolGraph.Root]
	Line       int            // 1-indexed
}

CallEdge represents a call relationship between two symbols. Resolution makes the extraction state explicit: 

CallResolutionResolved     — extractor identified CalleeID
CallResolutionUnresolved   — extractor attempted resolution and failed
CallResolutionNotAttempted — extractor recorded a heuristic/raw call only

func (CallEdge) Validate 

func (e CallEdge) Validate() error

Validate checks the defensive invariants a store must enforce before persisting an edge.

type CallResolution 

type CallResolution string

CallResolution records how much work an extractor did to identify a call's target. The zero value is invalid; use one of the constants below. 

const (
	CallResolutionResolved     CallResolution = "resolved"
	CallResolutionUnresolved   CallResolution = "unresolved"
	CallResolutionNotAttempted CallResolution = "not_attempted"
)

type CallSet 

type CallSet struct {
	Sites     []CallSite
	Truncated bool
}

CallSet is a bounded callers/callees result. Truncated reports whether more matching edges existed than the store returned.

type CallSite 

type CallSite struct {
	Symbol SymbolNode
	Edge   CallEdge
}

CallSite pairs a related symbol with the edge that connects it to the symbol the caller asked about. Returned by SymbolStore.Callers and SymbolStore.Callees so call-site File/Line are not lost in the lookup.

For Callers(target): Symbol is the caller, Edge.CalleeID == target. For Callees(target): Symbol is the callee, Edge.CallerID == target.

type Extractor 

type Extractor interface {
	// Languages returns the set of language identifiers this extractor can
	// produce graphs for, using rag.Chunk.Language tokens ("go", "python",
	// "typescript", ...). The returned slice MUST be safe for the caller
	// to read concurrently with future Extract calls.
	Languages() []string

	// Extract walks root and produces a graph stamped with scope and
	// vectorSpaceID. Scope is the caller-defined corpus/root/index namespace
	// used by SymbolStore to isolate graphs that share the same vector space.
	// An empty graph with nil error is the correct response for a tree
	// containing no sources in any of this extractor's languages — that
	// case is success, not failure. Errors are reserved for IO / parse
	// failures the caller cannot recover from.
	//
	// The returned [SymbolGraph.Root] MUST equal root after the same
	// canonicalization Extract applies to node files (filepath.Clean +
	// forward slashes), so paths join cleanly.
	// The returned [SymbolGraph.ExtractionSignature] is the opaque token
	// callers persist and later pass to Stale.
	Extract(ctx context.Context, scope string, root string, vectorSpaceID string) (SymbolGraph, error)

	// Stale reports whether a graph previously extracted from root and
	// scope, then stamped with prevSignature, can still be reused, or whether
	// the caller must re-extract. The signature is an opaque token defined
	// by the extractor itself — it MAY encode the vector-space identity,
	// source-file fingerprints, the extractor's own version, the toolchain
	// version, or any combination. Callers treat it as a string and pass
	// it back unchanged.
	//
	// Returning (true, nil) forces re-extraction. Returning (false, nil)
	// authorizes reuse of the previously persisted graph.
	Stale(ctx context.Context, scope string, root string, prevSignature string) (bool, error)
}

Extractor walks a source tree and produces a SymbolGraph. The interface is language-agnostic so a single SymbolStore consumer can be wired to language-specific extractors (Go via go/ast, tree-sitter for others) without leaking parser types across the seam.

Implementations MUST be safe for concurrent use; a caller indexing many roots in parallel will invoke Extract from multiple goroutines.

type GoExtractor 

type GoExtractor struct{}

GoExtractor walks Go source trees with the standard library go/ast parser. The zero value is ready for use and safe for concurrent calls.

func (GoExtractor) Extract 

func (GoExtractor) Extract(ctx context.Context, scope string, root string, vectorSpaceID string) (SymbolGraph, error)

Extract implements Extractor.

func (GoExtractor) Languages 

func (GoExtractor) Languages() []string

Languages implements Extractor.

func (GoExtractor) Stale 

func (GoExtractor) Stale(ctx context.Context, scope string, root string, prevSignature string) (bool, error)

Stale implements Extractor.

type NoOpExtractor 

type NoOpExtractor struct{}

NoOpExtractor satisfies Extractor without doing any work. It is the safe choice for callers that have not yet wired a real extractor: Languages reports no supported languages, Extract returns an empty graph stamped with the requested scope/vsid, and Stale always reports true (so any previously-cached state is invalidated rather than served).

func (NoOpExtractor) Extract 

func (NoOpExtractor) Extract(_ context.Context, scope string, root string, vectorSpaceID string) (SymbolGraph, error)

Extract implements Extractor. Always succeeds with an empty graph stamped with the requested scope, vsid, and canonicalized root.

func (NoOpExtractor) Languages 

func (NoOpExtractor) Languages() []string

Languages implements Extractor. Returns nil — claiming no languages lets callers short-circuit on the Languages check before invoking Extract.

func (NoOpExtractor) Stale 

func (NoOpExtractor) Stale(_ context.Context, _ string, _ string, _ string) (bool, error)

Stale implements Extractor. Always returns true so a caller relying on a no-op extractor never accidentally serves a stale persisted graph; this is fail-closed by design and matches the safety stance the package's drift-signature scope inherits from the chunk store.

type NoOpStore 

type NoOpStore struct{}

NoOpStore satisfies SymbolStore without persisting anything. Writes succeed silently; reads return ErrSymbolNotFound (singular) or an empty CallSet (plural). Useful as the default when AST persistence has not been wired in by the consumer.

func (NoOpStore) Callees 

func (NoOpStore) Callees(_ context.Context, _ string, _ string, _ string, limit int) (CallSet, error)

Callees implements SymbolStore.

func (NoOpStore) Callers 

func (NoOpStore) Callers(_ context.Context, _ string, _ string, _ string, limit int) (CallSet, error)

Callers implements SymbolStore.

func (NoOpStore) DeleteGraph 

func (NoOpStore) DeleteGraph(_ context.Context, _ string, _ string) error

DeleteGraph implements SymbolStore.

func (NoOpStore) ExtractionSignature 

func (NoOpStore) ExtractionSignature(_ context.Context, _ string, _ string) (string, error)

ExtractionSignature implements SymbolStore.

func (NoOpStore) GetSymbol 

func (NoOpStore) GetSymbol(_ context.Context, _ string, _ string, _ string) (SymbolNode, error)

GetSymbol implements SymbolStore.

func (NoOpStore) SymbolEnclosing 

func (NoOpStore) SymbolEnclosing(_ context.Context, _ string, _ string, _ string, _ int) (SymbolNode, error)

SymbolEnclosing implements SymbolStore.

func (NoOpStore) UpsertGraph 

func (NoOpStore) UpsertGraph(_ context.Context, graph SymbolGraph) error

UpsertGraph implements SymbolStore.

type SymbolGraph 

type SymbolGraph struct {
	Scope               string // caller-defined corpus/root/index namespace
	VectorSpaceID       string // embedding-space identity stamped on persisted rows
	ExtractionSignature string // opaque token persisted for Extractor.Stale
	Root                string // anchor for SymbolNode.File and CallEdge.File
	Nodes               []SymbolNode
	Calls               []CallEdge
}

SymbolGraph is the result of a single extraction pass — a transport structure between an Extractor and a SymbolStore. Callers consume graphs by handing them to a store, not by traversal.

Root anchors all File fields in Nodes and Calls. Paths inside the graph are filepath.Clean'd with forward slashes, relative to Root.

type SymbolKey 

type SymbolKey struct {
	Language      string
	Kind          SymbolKind
	Namespace     string
	Receiver      string
	Name          string
	Disambiguator string
}

SymbolKey is the language-neutral identity material encoded by SymbolID. Namespace is the language's package/module/scope path. Disambiguator is an optional, language-specific stable suffix for overloads, local functions, anonymous functions, or any construct that cannot be uniquely named by the other fields alone. Go extractors normally leave Disambiguator empty.

type SymbolKind 

type SymbolKind string

SymbolKind is the structural category of a SymbolNode. Typed string (not iota) so the value survives JSON round-trips without a custom marshaler. 

const (
	SymbolKindUnknown   SymbolKind = "unknown"
	SymbolKindFunction  SymbolKind = "function"
	SymbolKindMethod    SymbolKind = "method"
	SymbolKindStruct    SymbolKind = "struct"
	SymbolKindInterface SymbolKind = "interface"
	SymbolKindVar       SymbolKind = "var"
	SymbolKindConst     SymbolKind = "const"
	SymbolKindType      SymbolKind = "type"
)

type SymbolNode 

type SymbolNode struct {
	ID            string
	Language      string     // rag.Chunk.Language token (e.g. "go", "python")
	Kind          SymbolKind // structural category
	Namespace     string     // package/module/scope path in the source language
	Name          string     // bare identifier (e.g. "ValidateToken")
	Receiver      string     // method receiver/class/type, empty for non-methods
	Disambiguator string     // optional stable suffix for overloads/local symbols
	File          string     // canonicalized per [SymbolGraph.Root]
	StartLine     int        // 1-indexed
	EndLine       int        // 1-indexed, inclusive
	Declaration   string     // formatted declaration/signature/type definition (raw)
	Doc           string     // associated source doc/comment if any (raw)
}

SymbolNode is a single resolved program entity. ID is the stable identity used as the endpoint of every edge in the graph; see SymbolID. The node does NOT carry its Scope or VectorSpaceID — the enclosing SymbolGraph owns those for the whole batch.

Declaration and Doc contain raw, unsanitized source content; see the package-level "Trust boundary" note.

type SymbolStore 

type SymbolStore interface {
	// UpsertGraph atomically replaces every row previously written under
	// (graph.Scope, graph.VectorSpaceID) with the contents of graph, including
	// graph.ExtractionSignature. Implementations MUST be transactional and MUST
	// reject invalid call edges with an error wrapping [ErrInvalidGraph].
	UpsertGraph(ctx context.Context, graph SymbolGraph) error

	// DeleteGraph removes every row stamped with (scope, vectorSpaceID). Used
	// for "drop without replacement" workflows (vsid retirement, migration
	// cleanup) — NOT as a precursor to UpsertGraph, which already replaces
	// atomically.
	DeleteGraph(ctx context.Context, scope string, vectorSpaceID string) error

	// ExtractionSignature returns the opaque signature persisted with the
	// current graph for (scope, vectorSpaceID). Callers pass this value back to
	// Extractor.Stale. Returns [ErrSymbolNotFound] when no graph is recorded.
	ExtractionSignature(ctx context.Context, scope string, vectorSpaceID string) (string, error)

	// GetSymbol resolves a SymbolID to its node within (scope, vectorSpaceID).
	// Returns [ErrSymbolNotFound] when the ID is not present in the
	// store for that scope/vsid.
	GetSymbol(ctx context.Context, scope string, vectorSpaceID string, symbolID string) (SymbolNode, error)

	// SymbolEnclosing returns the node whose [StartLine, EndLine] range
	// contains line, within file (canonicalized the same way the indexer
	// canonicalizes SymbolNode.File). Returns [ErrSymbolNotFound] when
	// no enclosing symbol is recorded.
	SymbolEnclosing(ctx context.Context, scope string, vectorSpaceID string, file string, line int) (SymbolNode, error)

	// Callers returns up to limit symbols that call symbolID, with the
	// edge that connects each one to symbolID. A limit of 0 lets the store use
	// its defensive default cap. Negative limits are invalid and MUST return
	// an error. CallSet.Truncated reports whether more matches existed than
	// were returned.
	Callers(ctx context.Context, scope string, vectorSpaceID string, symbolID string, limit int) (CallSet, error)

	// Callees returns up to limit symbols called by symbolID, with the
	// edge that connects symbolID to each one. Same limit semantics as
	// Callers.
	Callees(ctx context.Context, scope string, vectorSpaceID string, symbolID string, limit int) (CallSet, error)
}

SymbolStore persists and serves symbol graphs. A single implementation handles both halves; the interface is intentionally NOT split into reader/writer because no real consumer has yet justified the split.

Every write carries Scope and VectorSpaceID via the SymbolGraph; every read filters by both explicitly. Scope isolates corpora/roots that share an embedder; vector-space changes invalidate the graph alongside the parallel chunk store within that scope.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.