extract

Documentation

Overview

Package extract emits typed semantic facts (symbols, references, imports, types, heritage, syntax edges) by parsing source files with tree-sitter.

D-01 hard invariants (DO NOT VIOLATE):

• This package MUST NOT import "github.com/agenthands/helix/internal/repomap".
• This package MUST NOT contain init() registration of providers.
• The shared *treesitter.GrammarRegistry is INJECTED by the daemon (Phase 49 BUG-04 / EXTRACT-05 invariant).

Index

• Constants
• func CanonicalizeStableSymbolKey(k StableSymbolKey) string
• func StableSymbolID(k StableSymbolKey) semantic.SymbolID
• type ExtractFunc
• type ExtractedFile
  • func PartialExtract(file SourceFile, reason PartialReason, err error) *ExtractedFile
• type ExtractionStatus
• type FileFact
• type FileSemanticAvailability
• type HeritageFact
• type ImportFact
• type PartialReason
• type Position
• type Provider
• type Range
• type ReceiverFact
• type ReferenceFact
• type ReferenceKind
• type Registry
  • func NewExtractorRegistry(grammars *treesitter.GrammarRegistry, providers ...Provider) *Registry
  • func (r *Registry) Grammars() *treesitter.GrammarRegistry
  • func (r *Registry) Languages() []string
  • func (r *Registry) Provider(lang string) (Provider, bool)
• type SourceFile
• type StableSymbolKey
  • func BuildProviderKey(meta SymbolMeta) StableSymbolKey
• type SymbolFact
• type SymbolKind
• type SymbolMeta
• type TypeFact

Constants

const (
	ConfidenceLSPOnly     float32 = 1.00
	ConfidenceLSPMerged   float32 = 0.95
	ConfidenceTSPlusLocal float32 = 0.80
	ConfidenceTSOnly      float32 = 0.70
	ConfidenceHeuristic   float32 = 0.45
)

Confidence ladder per SPEC §11.2 — closed enum with float32 labels. Phase 59 emits exactly ConfidenceTSOnly. Phase 61's LSP-enrichment worker raises emitted facts to ConfidenceTSPlusLocal / ConfidenceLSPMerged / ConfidenceLSPOnly per the merge rule.

NOT to be confused with the 7-rung §38.2 type-resolution ladder (Phase 62 territory: 1.00/0.90/0.80/0.70/0.60/0.45/0.20).

Functions

func CanonicalizeStableSymbolKey

func CanonicalizeStableSymbolKey(k StableSymbolKey) string

CanonicalizeStableSymbolKey joins fields with NUL bytes in the SPEC §11.1 order. Field order is normative — never reorder; new fields append only.

func StableSymbolID

func StableSymbolID(k StableSymbolKey) semantic.SymbolID

StableSymbolID returns xxhash64 of the canonicalized key. SPEC §11.1.

Types

type ExtractFunc

type ExtractFunc func(ctx context.Context, p Provider, source []byte, file SourceFile) (*ExtractedFile, error)

ExtractFunc is the shared signature per-language provider packages (Phase 59 P04) implement. The function is documented here so the scheduler (P03) and provider packages converge on a single shape.

Implementations are responsible for:

• parsing source bytes via p.TreeSitterLanguage(),
• running the provider's compiled Queries against the parse tree,
• building SymbolFact / ReferenceFact / ImportFact / TypeFact / HeritageFact rows with confidence stamped to ConfidenceTSOnly,
• constructing the FileFact envelope with the right ExtractionStatus + PartialReason on parse / query errors.

Per-extraction QueryCursor objects MUST be Close()'d (Pitfall #2 in 59-RESEARCH.md). Provider-cached *tree_sitter.Query objects live for the daemon lifetime and are Close()'d on shutdown.

type ExtractedFile

type ExtractedFile struct {
	File          FileFact
	Symbols       []SymbolFact
	References    []ReferenceFact
	Imports       []ImportFact
	Types         []TypeFact
	Heritage      []HeritageFact
	Partial       bool
	PartialReason string
}

ExtractedFile bundles the per-file extraction output. The provider's ExtractFile (in P04 per-language packages) returns one of these per successful or partial extraction.

func PartialExtract

func PartialExtract(file SourceFile, reason PartialReason, err error) *ExtractedFile

PartialExtract builds an ExtractedFile representing a partial or unsupported extraction outcome. Used by:

• The Phase 59 P03 scheduler when a file's classified language returns "other" (D-05 unsupported_language path — file row only, no symbols/refs/imports written).
• Per-language providers when parse_error / query_error / timeout / file_too_large / binary_or_generated / permission_denied / extractor_bug fires (D-05 first-class-with-errors path — partial facts may still be written by the caller; this helper produces the FileFact envelope).

`reason` MUST be one of the closed-enum PartialReason values (CONTEXT.md D-05). Unsupported-language and file-too-large and binary-or-generated outcomes set ExtractionStatus = ExtractionStatusUnsupported (the file was seen but cannot produce facts). All other reasons set ExtractionStatus = ExtractionStatusPartial (partial facts may have been written).

type ExtractionStatus

type ExtractionStatus string

ExtractionStatus is the per-file extraction-outcome enum (D-05).

const (
	ExtractionStatusReady       ExtractionStatus = "ready"
	ExtractionStatusPartial     ExtractionStatus = "partial"
	ExtractionStatusUnsupported ExtractionStatus = "unsupported"
	ExtractionStatusFailed      ExtractionStatus = "failed"
)

type FileFact

type FileFact struct {
	Path              string
	Language          string
	ExtractionStatus  ExtractionStatus
	ExtractionPartial bool
	PartialReason     PartialReason
	ExtractorName     string
	ExtractorVersion  string
	ErrorMessage      string
}

FileFact is the per-file extraction outcome row (D-05). Mirrors the semantic_files columns added by P01's applyMigration002.

type FileSemanticAvailability

type FileSemanticAvailability string

FileSemanticAvailability is the consumer-side classification (D-05). Mirrors ExtractionStatus plus a "missing" rung for files that were never extracted.

const (
	AvailabilityReady       FileSemanticAvailability = "ready"
	AvailabilityPartial     FileSemanticAvailability = "partial"
	AvailabilityUnsupported FileSemanticAvailability = "unsupported"
	AvailabilityFailed      FileSemanticAvailability = "failed"
	AvailabilityMissing     FileSemanticAvailability = "missing"
)

type HeritageFact

type HeritageFact struct {
	ID        semantic.HeritageID
	Language  string
	SubjectID semantic.SymbolID
	Relation  string // "extends"|"implements"|"embeds"
	Target    string // target qualified name (resolution is later)
	Range     Range
}

HeritageFact carries an extends/implements/embeds edge (D-01b).

type ImportFact

type ImportFact struct {
	ID       semantic.ImportID
	Language string
	Source   string // module path / package name
	Alias    string
	Symbols  []string // for explicit named imports
	File     string
	Range    Range
}

ImportFact is the in-memory shape per CONTEXT.md D-01b.

type PartialReason

type PartialReason string

PartialReason classifies why a file extraction landed in partial / unsupported / failed state. Closed enum.

const (
	PartialReasonUnsupportedLanguage PartialReason = "unsupported_language"
	PartialReasonParseError          PartialReason = "parse_error"
	PartialReasonQueryError          PartialReason = "query_error"
	PartialReasonTimeout             PartialReason = "timeout"
	PartialReasonFileTooLarge        PartialReason = "file_too_large"
	PartialReasonBinaryOrGenerated   PartialReason = "binary_or_generated"
	PartialReasonPermissionDenied    PartialReason = "permission_denied"
	PartialReasonExtractorBug        PartialReason = "extractor_bug"
)

type Position

type Position struct {
	Line   uint32
	Column uint32
}

Position is a (zero-based line, zero-based column UTF-16 code-unit) pair. Phase 59 keeps this tiny and local rather than re-using protocol/gen LSP types so internal/semantic/extract has no LSP coupling.

type Provider

type Provider interface {
	// Language returns the canonical language identifier (e.g. "go",
	// "typescript", "python"). Used as the Registry key. MUST be stable
	// across daemon restarts.
	Language() string

	// Extensions lists the file extensions claimed by this provider
	// (e.g. ".go" or ".ts"). Used by the scheduler to dispatch source
	// files to the right provider when language is not pre-classified.
	Extensions() []string

	// TreeSitterLanguage returns the tree-sitter language pointer the
	// provider parses with. The pointer is owned by the daemon-injected
	// *treesitter.GrammarRegistry singleton (BUG-04 invariant); the
	// provider does NOT construct or cache its own grammar registry.
	TreeSitterLanguage() *tree_sitter.Language

	// Queries returns the raw embedded queries.scm text for this
	// language. Compilation to *tree_sitter.Query is done once at
	// provider construction time and cached on the concrete struct
	// (Pitfall #2 in 59-RESEARCH.md).
	Queries() string

	// SupportsLSPEnrichment indicates whether Phase 61's enrichment
	// worker may attempt to merge this language's facts with LSP
	// callbacks. Languages whose LSP coverage is unstable can return
	// false to opt out.
	SupportsLSPEnrichment() bool
}

Provider mirrors SPEC §13.3 LanguageProvider — the surface a per-language extraction provider exposes to the daemon-owned Registry.

Phase 59 Wave 1 freezes the lookup-side surface (Language, Extensions, TreeSitterLanguage, Queries, SupportsLSPEnrichment). Per-language helper signatures (ImportResolver, ScopeBuilder, SymbolNormalizer, ReferenceClassifier) are intentionally NOT here yet — they are finalized in Phase 59 P04 when the per-language providers land. Defining them before they are needed risks lock-in to incorrect shapes.

type Range

type Range struct {
	Start Position
	End   Position
}

Range is a half-open [Start, End) source-range pair.

type ReceiverFact

type ReceiverFact struct {
	Type string
	Name string
}

ReceiverFact carries method-receiver metadata (Go-style methods, Python instance methods). Nil on free functions.

type ReferenceFact

type ReferenceFact struct {
	ID               semantic.ReferenceID
	Language         string
	Kind             ReferenceKind // CALL|REFERENCES|USES_TYPE|READS|WRITES|...
	Name             string
	File             string
	Range            Range
	ContainerID      *semantic.SymbolID
	ReceiverText     string             // pre-resolution literal, if any
	ResolvedTarget   *semantic.SymbolID // unset in Phase 59; Phase 61 fills via LSP
	ResolutionSource string             // "" until enrichment lands
	ValidationState  string             // "syntactic" baseline in Phase 59
	Confidence       float32
	Reason           string
	Partial          bool
	PartialReason    string
}

ReferenceFact is the in-memory shape per CONTEXT.md D-01b.

type ReferenceKind

type ReferenceKind string

ReferenceKind enumerates the reference categories Phase 59 emits. (CALL|REFERENCES|USES_TYPE|READS|WRITES|...). Concrete values land alongside per-language providers in Phase 59 P04; the type alias is the load-bearing contract.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry is the daemon-owned catalogue of per-language Provider implementations. Keyed by Provider.Language(). Constructed exactly once in internal/daemon/daemon.go (Phase 59 P05) — never via init() (D-02).

func NewExtractorRegistry

func NewExtractorRegistry(grammars *treesitter.GrammarRegistry, providers ...Provider) *Registry

NewExtractorRegistry constructs a Registry from explicitly-passed providers. The daemon owns the singleton *treesitter.GrammarRegistry (BUG-04 invariant) and injects it here.

Panics on:

• nil GrammarRegistry — wiring bug (T-59-02-01 mitigation).
• duplicate Provider.Language() — wiring bug (T-59-02-01 mitigation).

Both are start-time conditions, never runtime; surfacing as panics makes the wiring bug fail-fast at daemon bootstrap.

func (*Registry) Grammars

func (r *Registry) Grammars() *treesitter.GrammarRegistry

Grammars returns the daemon-injected *treesitter.GrammarRegistry the Registry was constructed with. Provider implementations call this to resolve their tree-sitter language pointer instead of constructing their own grammar registry (BUG-04 invariant).

func (*Registry) Languages

func (r *Registry) Languages() []string

Languages returns the registered language identifiers. Order is not guaranteed; callers that need stable ordering must sort.

func (*Registry) Provider

func (r *Registry) Provider(lang string) (Provider, bool)

Provider returns the registered Provider for the given language and true if registered, nil and false otherwise.

type SourceFile

type SourceFile struct {
	Path     string
	Language string
}

SourceFile is the provider-input envelope. The scheduler hands the provider (path, language) so the provider can build FileFact / ExtractedFile without re-classifying language by extension.

type StableSymbolKey

type StableSymbolKey struct {
	RepoID           string
	Language         string
	PackagePath      string
	OwnerPath        string
	QualifiedName    string
	Kind             string
	SignatureHash    string
	LSPIdentity      string
	FilePathFallback string
}

StableSymbolKey carries the 9 fields fed into stable-symbol-ID canonicalization per SPEC §11.1. Field order is normative — never reorder; new fields append only.

func BuildProviderKey

func BuildProviderKey(meta SymbolMeta) StableSymbolKey

BuildProviderKey constructs a StableSymbolKey from extracted symbol metadata. EXTRACT-02's same-content-rename invariant lives HERE (not in StableSymbolID itself):

• For exported symbols (Visibility == "exported") whose QualifiedName is stable across file moves within the same package, FilePathFallback is left empty so renaming the source file does NOT churn the ID.
• For unexported symbols (where QualifiedName collisions across files in the same package are possible), FilePathFallback is set to the relative file path so the ID disambiguates.

Callers (P04 per-language providers) MUST use BuildProviderKey rather than constructing StableSymbolKey directly.

LSPIdentity is intentionally left empty in Phase 59. Phase 61 may populate it during LSP enrichment, at which point the canonicalized input changes and the ID will too — by design (see SPEC §11.1 rule 1).

type SymbolFact

type SymbolFact struct {
	ID               semantic.SymbolID // xxhash64(CanonicalizeStableSymbolKey)
	StableKey        StableSymbolKey
	Language         string
	Kind             SymbolKind
	Name             string
	QualifiedName    string
	File             string
	Range            Range
	SelectionRange   Range
	ContainerID      *semantic.SymbolID // owner symbol (e.g., enclosing class)
	Signature        string
	SignatureHash    string
	Receiver         *ReceiverFact // Go methods, Python instance methods
	Visibility       string        // exported|private|package|...
	Doc              string        // leading-comment block, if extracted
	Confidence       float32       // 0.70 ts-only baseline; raised on merge
	ExtractionSource string        // "tree_sitter" in this phase
	Partial          bool
	PartialReason    string
}

SymbolFact is the in-memory shape per CONTEXT.md D-01b. Persistence mapping to DuckDB columns lives in internal/semantic/store/.

type SymbolKind

type SymbolKind string

SymbolKind enumerates the symbol categories Phase 59's tree-sitter providers emit. Closed enum (D-01b): consumers may rely on exhaustive case discrimination.

const (
	KindFunction  SymbolKind = "function"
	KindMethod    SymbolKind = "method"
	KindStruct    SymbolKind = "struct"
	KindClass     SymbolKind = "class"
	KindInterface SymbolKind = "interface"
	KindEnum      SymbolKind = "enum"
	KindType      SymbolKind = "type"
	KindVariable  SymbolKind = "variable"
	KindConstant  SymbolKind = "constant"
	KindParameter SymbolKind = "parameter"
	KindField     SymbolKind = "field"
)

type SymbolMeta

type SymbolMeta struct {
	RepoID, Language, PackagePath, OwnerPath, QualifiedName string
	Kind, SignatureHash, RelPath, Visibility                string
}

SymbolMeta is the input shape consumed by BuildProviderKey. Per-language providers (P04) assemble a SymbolMeta from tree-sitter capture results and call BuildProviderKey to obtain a StableSymbolKey for hashing.

type TypeFact

type TypeFact struct {
	ID         semantic.TypeFactID
	Language   string
	SubjectID  semantic.SymbolID // symbol the annotation is attached to
	Annotation string            // raw annotation text
	Range      Range
}

TypeFact carries a type-annotation fact attached to a symbol (D-01b).