retriever

Documentation

Overview

Package retriever generates candidate pairs for the expensive structural comparison stage. It treats candidate generation as an information-retrieval problem: three cheap channels — structural shape, ontology concepts, and resolved calls — each retrieve per-function top-K neighbors weighted by corpus rarity, and the union goes to the comparator. The first-stage number is therefore evidence mass (how much rare, informative material two functions share), not duplicate confidence: a perfect match on a ubiquitous trivial shape carries almost no evidence, while a moderate match supported by rare shingles, rare tags, or rare callees carries a lot.

Index

• Constants
• func Probe(units []parser.CodeUnit, probeIdx int, g *concepter.Graph, ...) ([]Candidate, Stats)
• func Retrieve(units []parser.CodeUnit, g *concepter.Graph, onto *ontology.Ontology, ...) ([]Candidate, Stats)
• type Candidate
• type Options
  • func DefaultOptions() Options
• type SharedPattern
• type Stats

Constants

const (
	ChannelShape   = "shape"
	ChannelConcept = "concept"
	ChannelCall    = "call"
)

Channel names, in the fixed order they appear in Candidate.Channels.

Functions

func Probe

func Probe(units []parser.CodeUnit, probeIdx int, g *concepter.Graph,
	onto *ontology.Ontology, ic *ontology.IC, opt Options) ([]Candidate, Stats)

Probe retrieves the corpus functions most related to units[probeIdx]. It runs the same three channels, the same gates, and the same definitive evidence arithmetic as Retrieve — only the admission loop is narrowed to the probe's turn, so a query costs index building plus one function's retrieval rather than the corpus's.

The probe must already be a member of units: every index is positional, and scoring a unit against statistics it is excluded from would misrepresent how it sits in this corpus. The caller appends it before tagging and graph building, which also hands it resolved callees for free.

func Retrieve

func Retrieve(units []parser.CodeUnit, g *concepter.Graph,
	onto *ontology.Ontology, ic *ontology.IC, opt Options) ([]Candidate, Stats)

Retrieve runs all three channels over the corpus and returns the deduped union with definitive per-pair evidence, sorted by (AIdx, BIdx). Ranking by evidence happens downstream, after the comparator — retriever output order is positional so the pipeline's positional doc lookup stays obvious.

Types

type Candidate

type Candidate struct {
	AIdx, BIdx int
	Breakdown  fingerprint.Breakdown // exact fingerprint similarity, always computed
	Shape      float64               // shared structural energy, Σ IC·min(count) over shared patterns
	Concept    float64               // shared tag information, Σ IC(LCS) over the best matching
	Call       float64               // shared rare-call IDF mass
	Total      float64               // Shape + Concept + Call, summed in that order
	TrophicSim float64               // 2·SharedEnergy/(E_A+E_B): weighted Dice over pattern energy
	CallSim    float64               // call-channel Dice: mutual fraction of informative call energy
	Channels   []string              // admission provenance, subset of {shape, concept, call}
	Chains     []SharedPattern       // highest-energy shared structures, the explanation
}

Candidate is one retrieved pair. AIdx < BIdx always; both index the units slice passed to Retrieve. The three evidence fields are Σ ln(N/df) over the shared rare features of each channel — nats of log-evidence over the same corpus, which is what makes summing them into Total coherent. Evidence is computed definitively for every union pair regardless of which channel admitted it, so a call-admitted pair still reports its shape mass.

type Options

type Options struct {
	ChannelK     int     // per-function, per-channel top-K
	Threshold    float64 // structural-channel floor on the exact fingerprint score
	MinNodes     int     // structural-channel eligibility gate on Fingerprint.Nodes
	MaxPatternDF int     // structural patterns present in more units than this carry no evidence
	MaxCallDF    int     // call tokens present in more units than this carry no evidence
	MaxConceptDF int     // concept postings larger than this are skipped for enumeration
	ChainTopN    int     // shared-structure explanations kept per pair
}

Options tunes retrieval. The df caps are not exposed as flags — they exist on Options so tests can shrink them without needing 50+-function fixtures.

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns the production defaults. ChannelK mirrors the --channel-k flag default; the caps are fixed constants chosen so that corpus-wide idioms (Error() shapes, fmt.Sprintf) drop out of the indexes entirely while genuinely shared machinery stays in.

type SharedPattern

type SharedPattern struct {
	Level  int
	Energy float64 // idf · min(count)
	Render string
}

SharedPattern is one shared high-level structure between two functions — the explanation of where their shared energy comes from.

type Stats

type Stats struct {
	ShapePairs        int // distinct pairs admitted by the structural channel
	ConceptPairs      int // distinct pairs admitted by the concept channel
	CallPairs         int // distinct pairs admitted by the call channel
	Union             int // unique pairs across all channels
	OnlyConcept       int // pairs only the concept channel admitted
	OnlyCall          int // pairs only the call channel admitted
	Suppressed        int // shape-eligible units whose every pattern was df-capped out
	LargeBuckets      int // exact pattern-multiset identity buckets with > largeBucketSize members
	SurvivingPatterns int // distinct structural patterns carrying evidence
}

Stats describes one retrieval run, for the stderr summary and evaluation.