go-lantern

README

go-lantern

Retrieval primitives for agent memory.

An agent has more memory than fits in a turn. Lantern takes a query and a corpus and returns the ranked few worth loading: BM25 for the words, embeddings for the meaning, and a fusion of the two that survives either one being unavailable.

It is a library, not a service. No storage, no scheduler, no runtime. It reads no environment variables and builds no HTTP client, so what a caller configures is what the caller gets.

Install

go get github.com/mirkobrombin/go-lantern@latest

Packages live under pkg/:

import "github.com/mirkobrombin/go-lantern/pkg/lexical"

Packages

package	what it does
lexical	Okapi BM25 ranking, script-agnostic tokenizer
embed	embedding providers, vector codec, cosine similarity
rank	score fusion, top-K, thresholds, near-duplicate collapse
health	provider probe, backfill contracts

lexical has no network and no configuration. It is the path that keeps working when a provider is down.

Lexical only

docs := []lexical.Doc{
    {ID: "a", Text: "the write ahead log survives a crash mid commit"},
    {ID: "b", Text: "the sidebar collapses below the medium breakpoint"},
}

for _, hit := range lexical.Rank("crash recovery", docs, lexical.Default()) {
    fmt.Println(hit.ID, hit.Score)
}

Ranking is deterministic: ties keep the input order, so the same corpus and query always produce the same list.

Hybrid

client := &http.Client{Timeout: 30 * time.Second}
provider, err := embed.NewOllama(embed.Config{
    BaseURL: embed.OllamaDefaultBaseURL,
    Model:   embed.OllamaDefaultModel,
    HTTP:    client,
})

Config.HTTP accepts anything with a Do method, so retry, circuit breaking and logging come from the caller's own client. Requests carry GetBody, which a retrying client needs to replay a body it already sent.

Score each candidate on both sides, then fuse:

results := []rank.Result{
    {ID: "a", Text: docA, Lexical: bm25A, Semantic: cosineA},
    {ID: "b", Text: docB, Lexical: bm25B, Semantic: cosineB},
}

fused := rank.Fuse(results, rank.DefaultWeights())
top := rank.TopK(rank.Dedupe(fused, rank.DefaultDuplicateThreshold), rank.DefaultMaxHits)

Each side is normalized across the candidate set before weighting. BM25 is unbounded and cosine is not, so combining them raw lets a long corpus drown the semantic score.

When no provider answered, the semantic column is flat and Fuse says so through SemanticApplied. The degraded mode is a weight set, rank.LexicalOnly(), and runs through the same fusion rather than a second code path.

Failure modes

Embedding failures are distinct because the reaction differs:

error	what to do
ErrProviderUnavailable	fall back to lexical, keep serving
ErrModelMissing	surface to an operator, retrying will not help
ErrTimeout	retry with backoff
ErrInvalidResponse	treat as unavailable
DimensionError	stop: vectors from two models are being compared

Cosine returns DimensionError rather than a zero score on a length mismatch, and Unpack reports a truncated blob rather than a shorter vector. Both would otherwise produce plausible rankings from broken data.

Store ModelInfo.Namespace() with every vector. A model swap invalidates every comparison made against what came before, and the namespace is what makes that detectable.

Backfill

health.Backfill embeds the records a Source reports as pending and writes them through a Sink. Its Report does not pass as success when it is partial:

report, err := health.Backfill(ctx, provider, src, dst, 100)
if err != nil {
    return err
}
if err := report.Err(); err != nil {
    return err // some records still have no vector
}

A half-filled corpus answers every query from the lexical path alone, silently, until someone notices the results got worse.

License

MIT