memory

package module
v1.5.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 24 Imported by: 0

README

memory

ci Go Reference

Hierarchical agent memory for Go — a portable library for durable, searchable memory entries with optional vector search and temporal APIs.

Module: github.com/iome-sh/memory

Features

  • File-backed store — atomic writes, tiers (working / contextual / semantic / archival), caller-managed best-effort version snapshots (overwrite does not auto-increment)
  • Hybrid search — keyword + optional dense/sparse vectors (Qdrant) and multi-factor re-ranking
  • Temporal APIs — session/time filters, event-time timelines, as-of fact listing, supersession helpers
  • Multi-hop retrieval — lightweight entity-graph expansion with hop-distance ranking
  • Pluggable embeddings — deterministic hash default for tests; production ONNX via hugot (pure-Go GoMLX or optional ORT)
  • Compaction hooks — kernel primitives for recency/compaction pipelines
  • Benchmarks — LongMemEval-oriented tooling under cmd/ and scripts/

Install

go get github.com/iome-sh/memory@latest
# or pin a release: go get github.com/iome-sh/memory@v1.5.8

Requires the Go version in go.mod. CI uses GOTOOLCHAIN=auto.

Quick start

package main

import (
	"fmt"

	"github.com/iome-sh/memory"
)

func main() {
	store := memory.NewPalaceStore("./data/palace")

	id := memory.GenerateMemoryID()
	_ = store.Write(memory.MemoryEntry{
		ID:   id,
		Tier: memory.TierContextual,
		Content: memory.MemoryContent{
			Summary: "Project alpha ships on Friday",
		},
	})

	hits := store.SearchMemory("project alpha", nil, 5, nil)
	for _, h := range hits {
		fmt.Println(h.Content.Summary)
	}
}

If PalaceConfig.BaseDir (or NewPalaceStore's argument) is empty, the store uses .palace under the process working directory (DefaultPalaceBaseDir). Prefer an explicit path in applications. This is a local filesystem root — not a leftover .ossa product path and not a hosted palace.

This package is a local filesystem library, not a cloud multi-tenant service. It does not implement mesh X-IOMesh-Org headers. Isolation is the directory you pass as BaseDir (or OS isolation around that directory). dual_write OFF · not Memory GA.

Optional semantic embeddings
embedFn, err := memory.NewGONNXEmbeddingFuncFromEnv()
if err != nil {
	panic(err)
}
store := memory.NewPalaceStoreWithConfig(memory.PalaceConfig{
	BaseDir:       "./data/palace",
	EmbeddingFunc: embedFn,
})
Variable Purpose
MEMORY_ONNX_MODEL_PATH Hugot model directory or .onnx file
MEMORY_HUGOT_BACKEND go (default pure-Go), ort, or auto
MEMORY_ORT_LIBRARY_DIR Directory containing ONNX Runtime shared library
MEMORY_ORT_CUDA 1 to enable CUDA EP (Linux ORT builds)
MEMORY_EMBEDDING_STRICT true to disable hash fallback on inference errors

Default ONNX export is BGE-small-en-v1.5 (384 dimensions). When using Qdrant with that model, set collection EmbeddingDim to 384.

Download helper:

go run ./scripts/download_onnx_model.go
Optional Qdrant
podman run -d --name qdrant \
  -p 6333:6333 -p 6334:6334 \
  -v qdrant_storage:/qdrant/storage:z \
  qdrant/qdrant

store := memory.NewPalaceStoreWithConfig(memory.PalaceConfig{
	BaseDir:          "./data/palace",
	VectorURL:        "http://localhost:6333",
	VectorCollection: "memory_collection",
	EmbeddingFunc:    embedFn, // recommended for semantic recall
})

Unit tests run without Podman/Qdrant. Integration helpers start a temporary container when available; set PODMAN_QDRANT_SKIP=1 to force skip.

API overview

Area Entry points
Store NewPalaceStore, NewPalaceStoreWithConfig, Write, Read, …
Search SearchMemory, SearchMemoryWithOptions
Timeline ListMemoryWithOptions
As-of facts ListFactsAsOf, ParseValidityWindow, EntryValidAt
Supersession SupersedeEntityFacts, WriteAndSupersede
Multi-hop MultiHopRetrieve, ExpandRelatedEntities, ExpandRelatedEntitiesHops
Vectors NewVectorStore, collection create/upsert helpers
Embeddings GenerateSimpleEmbedding, NewGONNXEmbeddingFunc, NewGONNXEmbeddingFuncFromEnv
Search options
from := time.Now().Add(-24 * time.Hour)
results := store.SearchMemoryWithOptions("project goals", memory.SearchMemoryOptions{
	SessionID:      "sess-abc",
	TimeFrom:       &from,
	Limit:          10,
	ReRankTemporal: true,
})
Field Effect
SessionID Keep entries with matching session
TimeFrom / TimeTo Inclusive event-time window
Limit Cap results (default 10 for search)
Tier Optional tier filter
QueryVec Dense re-rank when non-empty; keyword token hits stay ahead of Limit
ReRankTemporal Sort by relevance after keyword/vector path; keyword hits stay ahead of Limit
Timeline list
timeline := store.ListMemoryWithOptions(memory.ListMemoryOptions{
	SessionID: "sess-abc",
	TimeFrom:  &from,
	TagPrefix: "subject:",
	Limit:     50,
})

Filters apply before Limit. Default limit is 50 when ≤ 0. Listing uses a best-effort in-memory meta index plus an optional durable snapshot (indexes/event-time.json). A clean index is patched on Write / unlink instead of walking every tier JSON; a new process skips re-parse when the stamp matches. FS Palace remains source of truth. DisableMetaIndex / DisableDurableIndex opt out. Btree/tag secondary indexes remain residual.

Full reference: pkg.go.dev/github.com/iome-sh/memory.

Development

git clone https://github.com/iome-sh/memory.git
cd memory
go mod download

make check   # fmt-check + vet + test
make ci      # + govulncheck + build
make test
make test-race   # optional

See CONTRIBUTING.md for the contributor guide and SECURITY.md for reporting vulnerabilities.

LongMemEval tooling (optional)

Offline overlap smoke/bench (no OpenAI). Printed aggregate recall is top-k gold-answer string overlap (judge-free). It is not official V1 evaluate_qa.py + gpt-4o accuracy and not V2 LAFS Gain. Hash embeddings are the no-dep default; do not publish hash overlap as a leaderboard number. dual_write stays OFF. Not Memory GA.

make longmemeval-smoke
make longmemeval-recall-gate
make longmemeval-bench
make longmemeval-v2-bench   # official V2 file layout; does not vendor the 7GB snapshot

Official V1 scored QA: make longmemeval-judge (needs OPENAI_API_KEY). Official V2 scored runs use the upstream harness with a fixed Qwen3.5-9B reader and GPT-5.2 judge — this kernel only loads V2 files and exposes Insert/Query. Full dataset / judge flows need extra deps and keys; see comments in Makefile and scripts/.

--limit N on scripts/longmemeval_qa_generate.py is dataset prefix order. Official V1 starts with temporal-reasoning, so a small n is not a mixed V1 score. Use --sample mixed (or LONGMEMEVAL_QA_SAMPLE=mixed) for a stratified slice and print the type histogram. Prefix-n is not overall V1. overlap ≠ gpt-4o ≠ V2 LAFS.

/retrieve accepts session_id (official generate passes conv_id / question_id). Shared-palace QA without it is other-session dominated. Hypothesis JSONL keeps question_date, retrieve snippets, and embed_mode for audit. Hash default. Not a leaderboard submit. Not Memory GA.

Haystack dates accept official cleaned 2006/01/02 (Mon) 15:04 as well as RFC3339.

Documentation

Document Description
CHANGELOG.md Release notes
RELEASING.md How maintainers tag module versions; support / version policy for consumers
SECURITY.md Security policy and supported-versions table
CONTRIBUTING.md Development workflow
CODE_OF_CONDUCT.md Community standards
SUPPORT.md How to get help; scope (library kernel) and related host
docs/temporal-memory-kernel-roadmap.md Temporal API roadmap (K0–K4 style)
docs/OPEN_SOURCE_AUDIT.md OSS process checklist
Repository Role
iomesh-memory-mcp Lean MCP host binary for this kernel
iomesh-tui Multi-provider agent TUI/CLI (optional mesh hooks)
iomesh-client-sdk-go Official Go client for I/O Mesh
iomesh-client-sdk-python Official Python client for I/O Mesh (Beta / pre-1.0 — not invent 1.0 / live PyPI GA)

This module is a library (tags for go get). Binary packaging, SBOM, and cosign apply to host tools such as iomesh-memory-mcp — see RELEASING.md.

License

MIT · NOTICE

Documentation

Index

Constants

View Source 
const (
	// MiniLMEmbeddingDim is the output size for all-MiniLM-L6-v2 (and KnightsAnalytics ONNX export).
	MiniLMEmbeddingDim = 384
	// MiniLMMaxSeqTokens is the BERT position limit for all-MiniLM-L6-v2 ONNX exports.
	MiniLMMaxSeqTokens = 512

	// DefaultHashEmbeddingDim is the dimension used by GenerateSimpleEmbedding when dim <= 0.
	DefaultHashEmbeddingDim = 768
	// EnvONNXModelPath points at a hugot model directory (tokenizer + model.onnx) or a single .onnx file.
	EnvONNXModelPath = "MEMORY_ONNX_MODEL_PATH"
	// EnvEmbeddingStrict disables hash fallback when ONNX inference fails (recommended in production).
	EnvEmbeddingStrict = "MEMORY_EMBEDDING_STRICT"
)
View Source 
const (
	// EnvHugotBackend selects the hugot inference backend: go (default), ort, or auto.
	EnvHugotBackend = "MEMORY_HUGOT_BACKEND"
	// EnvORTLibraryDir is the directory containing libonnxruntime.so/.dylib (ORT builds only).
	EnvORTLibraryDir = "MEMORY_ORT_LIBRARY_DIR"
	// EnvORTCuda enables the CUDA execution provider when set to 1/true.
	EnvORTCuda = "MEMORY_ORT_CUDA"
	// EnvORTCoreML enables the CoreML execution provider when set to 1/true.
	EnvORTCoreML = "MEMORY_ORT_COREML"
	// EnvORTCudaDeviceID selects the CUDA device (default 0).
	EnvORTCudaDeviceID = "MEMORY_ORT_CUDA_DEVICE_ID"
)
View Source 
const (
	// DefaultONNXModelHF is the recommended hugot ONNX export for agent memory (384-d, CPU ORT).
	DefaultONNXModelHF = "KnightsAnalytics/bge-small-en-v1.5"
	// LegacyONNXModelHF is the previous default (384-d MiniLM).
	LegacyONNXModelHF = "KnightsAnalytics/all-MiniLM-L6-v2"
	// BGESmallEmbeddingDim is the output width for BAAI/bge-small-en-v1.5 ONNX exports.
	BGESmallEmbeddingDim = 384
	// EnvEmbeddingModelHF overrides the Hugging Face model id used for documentation and download scripts.
	EnvEmbeddingModelHF = "MEMORY_EMBEDDING_MODEL"
)
View Source 
const DefaultPalaceBaseDir = ".palace"

DefaultPalaceBaseDir is the local palace root used when PalaceConfig.BaseDir (or NewPalaceStore's argument) is empty. Relative to the process working directory. Not a leftover product path and not a hosted palace.

Variables

View Source 
var DefaultCompactionConfig = CompactionConfig{
	Tier2Strategy:         StrategyPatternExtraction,
	Tier3Strategy:         StrategyCorePrinciple,
	TemporalWindowSize:    12,
	SimilarityThreshold:   0.75,
	DataSim:               0.7,
	DataCount:             5,
	ProtectHighScoreFacts: true,
	FactScoreThreshold:    0.90,
}

Functions

func CalculateRecencyBoost 

func CalculateRecencyBoost(deltaHours float64) float64

CalculateRecencyBoost returns recency boost factor

func CalculateRelevanceScore 

func CalculateRelevanceScore(entry MemoryEntry) float64

CalculateRelevanceScore combines score impact, recency, temporal decay, usage

func CalculateTemporalDecay 

func CalculateTemporalDecay(entry MemoryEntry) float64

CalculateTemporalDecay implements H-Mem style forgetting curve

func CosineSimilarity 

func CosineSimilarity(a, b []float32) float64

CosineSimilarity helper

func DefaultEmbeddingModelFromEnv 

func DefaultEmbeddingModelFromEnv() string

DefaultEmbeddingModelFromEnv returns the configured HF model id or DefaultONNXModelHF.

func DefaultONNXModelCacheDirName 

func DefaultONNXModelCacheDirName() string

DefaultONNXModelCacheDirName is the testdata/models subdirectory for the default ONNX export.

func EntryEntityKeys 

func EntryEntityKeys(e MemoryEntry) []string

EntryEntityKeys extracts entity-like keys from an entry for multi-hop matching. Sources:

  • TemporalTags with "entity:" prefix → value after prefix (e.g. entity:person:alice → person:alice)
  • TemporalTags with "subject:" prefix → full tag (subject:auth)
  • Content.Tags with entity:/subject: prefixes (same rules)
  • Relations.RelatedConcepts (trimmed non-empty strings as-is)

Duplicates are removed; order is stable (first-seen).

func EntryHasTag 

func EntryHasTag(e MemoryEntry, tag string) bool

EntryHasTag reports whether e has an exact tag match in TemporalTags or Content.Tags.

func EntryHasTagPrefix 

func EntryHasTagPrefix(e MemoryEntry, prefix string) bool

EntryHasTagPrefix reports whether e has a TemporalTags or Content.Tags entry with the given prefix.

func EntryValidAt 

func EntryValidAt(e MemoryEntry, asOf time.Time) bool

EntryValidAt reports whether e is considered valid at asOf.

Rules (bi-temporal lite — validity window on entries, not full dual clocks):

  1. Zero asOf is treated as time.Now().UTC().
  2. When valid_from / valid_until tags are present: - if valid_from is set and asOf.Before(from) → false - if valid_until is set and !asOf.Before(until) → false (valid_until is an exclusive end: asOf == until is invalid)
  3. When NO validity tags at all: fall back to "known by asOf" — valid if entryEventTime is zero OR !entryEventTime.After(asOf) (entry exists / was recorded by asOf).

This is not a full temporal knowledge graph (no transaction time, no edge validity). Hosts that write validity tags get windowed facts; untagged entries remain historically "known once recorded."

func ExtractAtomicFacts 

func ExtractAtomicFacts(entry MemoryEntry) []string

ExtractAtomicFacts extracts high-value personal facts from a memory entry.

func GenerateMemoryID 

func GenerateMemoryID() string

GenerateMemoryID uses cuid2

func GenerateSimpleEmbedding 

func GenerateSimpleEmbedding(text string, dim int) []float32

GenerateSimpleEmbedding (deterministic hash-based fallback)

func HugotCacheDirName 

func HugotCacheDirName(hfModel string) string

HugotCacheDirName returns the local directory name hugot uses for a Hugging Face model id.

func MultiFactorScore 

func MultiFactorScore(entry MemoryEntry, queryVec []float32) float64

MultiFactorScore implements full H-Mem scoring

func ParseValidityWindow 

func ParseValidityWindow(e MemoryEntry) (from, until *time.Time)

ParseValidityWindow reads valid_from / valid_until from TemporalTags. Missing or unparseable bounds are open-ended (nil).

Tag format: "valid_from:<RFC3339>" and "valid_until:<RFC3339>" (time.RFC3339 / time.RFC3339Nano accepted via time.Parse).

func PopulateTemporalTags 

func PopulateTemporalTags(cycle int) []string

PopulateTemporalTags (cycle-aware)

func ResolveEmbeddingDim 

func ResolveEmbeddingDim(modelPath string) int

ResolveEmbeddingDim returns the Qdrant/Palace vector width for the active embedding backend. When an ONNX model path is configured, infers dimension from the model directory name; otherwise hash (768).

func ResolveEmbeddingDimFromEnv 

func ResolveEmbeddingDimFromEnv() int

ResolveEmbeddingDimFromEnv reads MEMORY_ONNX_MODEL_PATH and returns the matching dimension.

Types

type BatchCollectionError 

type BatchCollectionError struct {
	Failures []CollectionCreateResult
}

func (*BatchCollectionError) Error 

func (e *BatchCollectionError) Error() string

func (*BatchCollectionError) Is 

func (e *BatchCollectionError) Is(target error) bool

func (*BatchCollectionError) Unwrap 

func (e *BatchCollectionError) Unwrap() []error

type BatchEmbeddingFunc 

type BatchEmbeddingFunc func(texts []string, dim int) ([][]float32, error)

BatchEmbeddingFunc embeds multiple texts in one forward pass when supported (e.g. GONNXEmbedder).

type CollectionCreateResult 

type CollectionCreateResult struct {
	Name  string
	Error error
}

type CompactionAction 

type CompactionAction struct {
	Action    string
	TargetIDs []string
	Reason    string
}

type CompactionConfig 

type CompactionConfig struct {
	Tier2Strategy      CompactionStrategy `json:"tier2_strategy"`
	Tier3Strategy      CompactionStrategy `json:"tier3_strategy"`
	LastEvaluatedCycle int                `json:"last_evaluated_cycle"`
	AvgScoreBefore     float64            `json:"avg_score_before"`
	AvgScoreAfter      float64            `json:"avg_score_after"`
	Improvement        float64            `json:"improvement"`
	// H-Mem inspired
	TemporalWindowSize  int     `json:"temporal_window_size"` // beta-like (e.g. cycles or months)
	SimilarityThreshold float64 `json:"similarity_threshold"` // alpha
	// RecMem phase-transition (Phase 1)
	DataSim   float64 `json:"data_sim"`   // geometric similarity radius (default 0.7)
	DataCount int     `json:"data_count"` // critical recurrence count (default 5)
	// LongMemEval production hardening
	ProtectHighScoreFacts bool    `json:"protect_high_score_facts"` // default true
	FactScoreThreshold    float64 `json:"fact_score_threshold"`     // default 0.90
}

CompactionConfig (extended with RecMem phase-transition parameters)

type CompactionStrategy 

type CompactionStrategy string

CompactionStrategy types (from ossa, kept for compatibility) 

const (
	StrategySimpleSummary     CompactionStrategy = "simple_summary"
	StrategyPatternExtraction CompactionStrategy = "pattern_extraction"
	StrategyCorePrinciple     CompactionStrategy = "core_principle"
)

type EmbeddingFunc 

type EmbeddingFunc func(text string, dim int) []float32

EmbeddingFunc is injectable for semantic embeddings (Phase 5.1)

func NewGONNXEmbeddingFunc 

func NewGONNXEmbeddingFunc(modelPath string) (EmbeddingFunc, error)

NewGONNXEmbeddingFunc returns a Palace-compatible EmbeddingFunc backed by pure-Go ONNX. modelPath may be empty to use GenerateSimpleEmbedding (dev/tests). modelPath may be a hugot model directory or a direct path to a single .onnx file.

func NewGONNXEmbeddingFuncFromEnv 

func NewGONNXEmbeddingFuncFromEnv() (EmbeddingFunc, error)

NewGONNXEmbeddingFuncFromEnv loads MEMORY_ONNX_MODEL_PATH when set; otherwise hash embeddings.

type EntityGraph 

type EntityGraph struct {
	Entities map[string][]string `json:"entities"`
}

EntityGraph for H-Mem KG integration (richer relational graph)

type FactsAsOfOptions 

type FactsAsOfOptions struct {
	// AsOf is the validity instant (zero = Now UTC).
	AsOf time.Time
	// Query optional case-insensitive substring on Summary / Full / OriginalText.
	Query string
	// SessionID, when non-empty, keeps only matching SessionID.
	SessionID string
	// Entity filters TemporalTags:
	//   - if Entity contains ':', exact match on "entity:<value>" (or Entity itself
	//     when it already has the "entity:" prefix);
	//   - otherwise any TemporalTag with prefix "entity:" that contains Entity.
	Entity string
	// Limit caps results (default 50 when <= 0).
	Limit int
	// Tier when non-nil: only that tier. When nil: Working+Contextual+Semantic
	// (exclude Archival unless IncludeArchival).
	Tier *MemoryTier
	// IncludeArchival, when true and Tier==nil, also includes Archival.
	IncludeArchival bool
}

FactsAsOfOptions configures as-of validity listing (K4 first slice / s616).

Filters apply before Limit so many invalid-at-asOf entries do not underfill.

type GONNXEmbedder 

type GONNXEmbedder struct {
	// contains filtered or unexported fields
}

GONNXEmbedder runs sentence embeddings via hugot FeatureExtractionPipeline. Call Close when done to release the hugot session.

func NewGONNXEmbedder 

func NewGONNXEmbedder(opts GONNXOptions) (*GONNXEmbedder, error)

NewGONNXEmbedder loads a hugot-compatible ONNX model directory and starts a feature-extraction pipeline.

func (*GONNXEmbedder) Backend 

func (e *GONNXEmbedder) Backend() string

Backend returns the active hugot backend label (gomlx, ort+cpu, ort+cuda:0, ort+coreml).

func (*GONNXEmbedder) BatchFunc 

func (e *GONNXEmbedder) BatchFunc() BatchEmbeddingFunc

BatchFunc returns a Palace-compatible batch embedding closure for SearchMemory batch scoring.

func (*GONNXEmbedder) Close 

func (e *GONNXEmbedder) Close() error

Close releases hugot session resources.

func (*GONNXEmbedder) Dimension 

func (e *GONNXEmbedder) Dimension() int

Dimension returns the native embedding width produced by the loaded ONNX model.

func (*GONNXEmbedder) Embed 

func (e *GONNXEmbedder) Embed(text string) ([]float32, error)

Embed returns an L2-normalized embedding for text.

func (*GONNXEmbedder) EmbedBatch 

func (e *GONNXEmbedder) EmbedBatch(texts []string) ([][]float32, error)

EmbedBatch returns L2-normalized embeddings for all texts in one pipeline forward pass.

func (*GONNXEmbedder) Func 

func (e *GONNXEmbedder) Func() EmbeddingFunc

Func returns an EmbeddingFunc that preserves the PalaceConfig injectable signature. The returned closure keeps the embedder alive; callers should not discard the embedder until all embedding work is finished (or use Func() only via NewGONNXEmbeddingFunc).

type GONNXOptions 

type GONNXOptions struct {
	ModelPath string
	// Strict disables silent fallback to GenerateSimpleEmbedding on inference errors.
	Strict bool
	// HugotBackend overrides MEMORY_HUGOT_BACKEND for this embedder (go, ort, auto).
	HugotBackend HugotBackendConfig
}

GONNXOptions configures a hugot ONNX embedder (GoMLX by default; ORT when built with -tags ORT).

type HugotBackendConfig 

type HugotBackendConfig struct {
	Backend    string // go, ort, auto
	LibraryDir string
	Cuda       bool
	CoreML     bool
	DeviceID   string
}

HugotBackendConfig holds resolved hugot session options from the environment.

func ResolveHugotBackendFromEnv 

func ResolveHugotBackendFromEnv() HugotBackendConfig

ResolveHugotBackendFromEnv reads MEMORY_HUGOT_BACKEND and ORT accelerator env vars.

func (HugotBackendConfig) HugotBackendLabel 

func (c HugotBackendConfig) HugotBackendLabel() string

HugotBackendLabel returns a log-friendly backend description.

type ListMemoryOptions 

type ListMemoryOptions struct {
	SessionID string
	// TimeFrom / TimeTo filter by entry event time (see entryEventTime). Both inclusive when set.
	TimeFrom *time.Time
	TimeTo   *time.Time
	// Tag exact-matches any of TemporalTags or Content.Tags.
	Tag string
	// TagPrefix matches tag strings with strings.HasPrefix (e.g. "subject:", "session_seq:").
	TagPrefix string
	// Query optional case-insensitive substring on Summary / Full / OriginalText.
	Query string
	// Limit caps results (default 50 when <= 0 — timeline-friendly; not search's 10).
	Limit int
	// Tier when non-nil: only that tier. When nil: Working+Contextual+Semantic
	// (exclude Archival by default to match aion MCP timeline).
	Tier *MemoryTier
	// IncludeArchival, when true and Tier==nil, also includes Archival.
	IncludeArchival bool
	// Ascending: false = newest first (default); true = oldest first.
	Ascending bool
}

ListMemoryOptions configures event-time ordered listing with optional filters (K2 / s611).

Filters are applied before Limit so windowed/session timelines do not underfill when many out-of-scope entries exist (same underfill class as SearchMemoryWithOptions K1).

Complexity: FS Palace is source of truth. When the best-effort meta index is enabled (default), ListMemoryWithOptions filters on lightweight entryMeta and loads full JSON only for survivors (s1066). Write/unlink patches a clean index in place (#63). A durable snapshot at indexes/event-time.json (#44) avoids re-parsing every tier JSON on a fresh process when the stamp matches. With DisableMetaIndex, falls back to O(n) full entry scan.

type MemoryContent 

type MemoryContent struct {
	Summary string   `json:"summary"`
	Full    string   `json:"full,omitempty"`
	Tags    []string `json:"tags,omitempty"`
}

MemoryContent holds summary/full/tags

type MemoryEntry 

type MemoryEntry struct {
	ID           string     `json:"id"`
	Type         string     `json:"type"`
	Tier         MemoryTier `json:"tier"`
	Version      int        `json:"version"`
	CreatedAt    time.Time  `json:"created_at"`
	UpdatedAt    time.Time  `json:"updated_at"`
	Cycle        int        `json:"cycle"`
	TemporalTags []string   `json:"temporal_tags,omitempty"`
	AccessCount  int        `json:"access_count"`
	LastAccessed time.Time  `json:"last_accessed,omitempty"`

	// === LongMemEval / turn-level granularity extensions (production implementation) ===
	TurnID         string    `json:"turn_id,omitempty"`         // explicit round/turn identifier
	SessionID      string    `json:"session_id,omitempty"`      // multi-session grouping
	Timestamp      time.Time `json:"timestamp,omitempty"`       // precise event time for temporal reasoning
	ExtractedFacts []string  `json:"extracted_facts,omitempty"` // fact-augmented for better recall
	Keyphrases     []string  `json:"keyphrases,omitempty"`      // keyphrase expansion for indexing
	OriginalText   string    `json:"original_text,omitempty"`   // raw turn text for provenance

	Content    MemoryContent    `json:"content"`
	Provenance MemoryProvenance `json:"provenance"`
	Metrics    MemoryMetrics    `json:"metrics"`
	Relations  MemoryRelations  `json:"relations"`
}

MemoryEntry is the core unit stored in the Palace. Extended for LongMemEval production readiness: explicit turn/session granularity + fact-augmented indexing.

type MemoryMetrics 

type MemoryMetrics struct {
	ScoreImpact  float64   `json:"score_impact,omitempty"`
	UsageCount   int       `json:"usage_count"`
	LastAccessed time.Time `json:"last_accessed,omitempty"`
}

MemoryMetrics for scoring and access

type MemoryProvenance 

type MemoryProvenance struct {
	SourceCycle int      `json:"source_cycle"`
	SourceStep  string   `json:"source_step,omitempty"`
	ParentIDs   []string `json:"parent_ids,omitempty"`
	ToolCalls   []string `json:"tool_calls,omitempty"`
}

MemoryProvenance tracks origin

type MemoryRelations 

type MemoryRelations struct {
	ImprovesUpon    []string `json:"improves_upon,omitempty"`
	RelatedConcepts []string `json:"related_concepts,omitempty"`
	Backlinks       []string `json:"backlinks,omitempty"`
}

MemoryRelations for graph links

type MemoryStats 

type MemoryStats struct {
	WorkingCount    int
	ContextualCount int
	ArchivalCount   int
	SemanticCount   int // Phase 3
	TotalEntries    int
	LastCompaction  time.Time
}

MemoryStats provides observability metrics

type MemoryTier 

type MemoryTier int

MemoryTier defines the three-tier hierarchical memory (inspired by ossa Palace + H-Mem ideas) 

const (
	TierWorking    MemoryTier = 1
	TierContextual MemoryTier = 2
	TierArchival   MemoryTier = 3
	// TierSemantic is used for high-fidelity atomic facts protected by RecMem Phase 3
	TierSemantic MemoryTier = 4
)

type MultiHopOptions 

type MultiHopOptions struct {
	// SeedEntity is the starting entity key (normalized). Prefer exact graph node keys.
	SeedEntity string
	// SeedQuery optional: if set, run SearchMemoryWithOptions with Query then derive
	// entity seeds from top hits' TemporalTags entity:* and RelatedConcepts
	// (combined with SeedEntity when both are set).
	SeedQuery string
	// MaxHops default 2; clamped to 1..4. Hop 0 is the seed itself.
	MaxHops int
	// Limit default 20; applied AFTER expansion + entry collect + filters.
	Limit int
	// SessionID, when non-empty, keeps only matching SessionID (before Limit).
	SessionID string
	// AsOf optional EntryValidAt filter (before Limit).
	AsOf *time.Time
	// Tier when non-nil: only that tier. When nil: Working+Contextual+Semantic
	// (exclude Archival unless IncludeArchival).
	Tier *MemoryTier
	// IncludeArchival, when true and Tier==nil, also includes Archival.
	IncludeArchival bool
	// QueryVec optional pass-through when seeding via SeedQuery search.
	QueryVec []float32
	// PreferShorterHops ranks by minimum BFS hop distance from seed (lower first),
	// then event time descending within the same hop. Default true (nil or true).
	// Set to a false pointer to opt out and use legacy seed-match-first sort.
	PreferShorterHops *bool
}

MultiHopOptions configures associative / multi-hop lite retrieval over the EntityGraph plus entry entity tags (A2 first slice / s619; hop ranking s1067; residual honesty pin s1278).

Honesty: competitive multi-hop lite — BFS over GetRelatedEntities and entry collect via TemporalTags / Content.Tags / RelatedConcepts. Not a full Zep / Graphiti knowledge graph. Hop-distance ranking is path-aware ranking lite (prefer shorter BFS hops), not typed-edge / embedding-guided path scoring. Not full graph RAG; not product Memory GA (kernel-only).

type PalaceConfig 

type PalaceConfig struct {
	// BaseDir is the local palace root. Empty uses DefaultPalaceBaseDir (".palace").
	BaseDir            string
	MaxWorkingEntries  int
	MaxWorkingAgeHours int
	CompactionConfig   CompactionConfig
	EmbeddingFunc      EmbeddingFunc      `json:"-"` // pluggable (Phase 5.1)
	BatchEmbeddingFunc BatchEmbeddingFunc `json:"-"` // optional ONNX batch path (Phase 5.1)
	// DisableMetaIndex forces ListMemoryWithOptions to use the full FS scan path
	// instead of the best-effort in-memory metadata index (K2 residual / s1066).
	// Default false (index enabled). Useful for parity tests.
	DisableMetaIndex bool
	// DisableDurableIndex skips load/save of indexes/event-time.json.
	// The in-memory meta index still runs unless DisableMetaIndex is set.
	// Default false (durable snapshot enabled). FS Palace remains source of truth.
	DisableDurableIndex bool
}

PalaceConfig for configurable PalaceStore creation (Phase 4.3)

type PalaceStore 

type PalaceStore struct {
	BaseDir string
	Config  PalaceConfig
	// contains filtered or unexported fields
}

PalaceStore provides file-backed hierarchical memory storage.

func NewPalaceStore 

func NewPalaceStore(baseDir string) *PalaceStore

NewPalaceStore is a convenience constructor (uses defaults). Empty baseDir defaults to DefaultPalaceBaseDir.

func NewPalaceStoreWithConfig 

func NewPalaceStoreWithConfig(cfg PalaceConfig) *PalaceStore

NewPalaceStoreWithConfig creates PalaceStore with full configuration (Phase 4.3). Empty BaseDir defaults to DefaultPalaceBaseDir (".palace") under the process cwd.

func (*PalaceStore) AddEntityRelationship 

func (ps *PalaceStore) AddEntityRelationship(entity, related string)

AddEntityRelationship adds normalized entity links (H-Mem style). Ensures BaseDir/relations exists before write (safe if ensureDirs was skipped or the directory was removed).

func (*PalaceStore) AutoRecMemCompaction 

func (ps *PalaceStore) AutoRecMemCompaction(generateFn func(prompt string) string, vectorCb VectorStoreCallback)

AutoRecMemCompaction is the production entry point for automatic RecMem formation.

func (*PalaceStore) EvictWorkingTier 

func (ps *PalaceStore) EvictWorkingTier(maxAgeHours int, maxCount int)

EvictWorkingTier performs age/size-based eviction from Working tier (Phase 4.2)

func (*PalaceStore) ExpandRelatedEntities 

func (ps *PalaceStore) ExpandRelatedEntities(seed string, maxHops int) []string

ExpandRelatedEntities BFS from seed over GetRelatedEntities up to maxHops (includes seed at hop 0). maxHops is clamped to 1..4.

Returns unique entity keys in BFS discovery order (seed first).

func (*PalaceStore) ExpandRelatedEntitiesHops 

func (ps *PalaceStore) ExpandRelatedEntitiesHops(seed string, maxHops int) map[string]int

ExpandRelatedEntitiesHops BFS from seed over GetRelatedEntities up to maxHops (includes seed at hop 0). maxHops is clamped to 1..4.

Returns entity key → minimum hop distance from seed. Empty seed → nil.

func (*PalaceStore) GetRelatedEntities 

func (ps *PalaceStore) GetRelatedEntities(entity string) []string

GetRelatedEntities returns related entities for graph traversal

func (*PalaceStore) GetStats 

func (ps *PalaceStore) GetStats() MemoryStats

GetStats returns observability metrics (Phase 1.2)

func (*PalaceStore) IngestTurn 

func (ps *PalaceStore) IngestTurn(turn MemoryEntry) error

IngestTurn is the primary production entry point for LongMemEval benchmark and ego online session processing. It writes the parent turn first, then each non-empty ExtractedFacts child as type turn_fact. When ExtractedFacts is empty, facts are auto-extracted from the turn text.

Child Content.Tags inherit the parent turn's Content.Tags (trimmed, de-duplicated) and always include the structural markers fact_augmented and from_turn. The kernel does not stamp longmemeval; callers that want that label (the LongMemEval harness under cmd/longmemeval-* and internal/longmemeval) pass it on the parent so children inherit it.

Fact-augmented children get valid_from stamped when unset; child Write errors are returned.

Partial persist is the contract: a child Write error does not roll back the parent or earlier facts already written. A non-nil error does not mean nothing persisted. Not all-or-nothing. Not Memory GA. dual_write OFF.

func (*PalaceStore) InvalidateMetaIndex 

func (ps *PalaceStore) InvalidateMetaIndex()

InvalidateMetaIndex is the exported test/debug hook for force-rebuild.

func (*PalaceStore) ListEntriesInTier 

func (ps *PalaceStore) ListEntriesInTier(tier MemoryTier) []MemoryEntry

ListEntriesInTier returns all entries in the given tier, sorted by relevance score (descending).

func (*PalaceStore) ListFactsAsOf 

func (ps *PalaceStore) ListFactsAsOf(opts FactsAsOfOptions) []MemoryEntry

ListFactsAsOf lists entries valid at AsOf with optional filters. Filters apply before Limit (underfill class, same as K1/K2).

Order of operations: collect candidates → session → entity → query → EntryValidAt(asOf) → sort (Semantic first, then event time desc) → limit.

Default tiers when Tier == nil: Working + Contextual + Semantic (+ Archival if IncludeArchival).

Honesty: bi-temporal lite (validity window tags). Not full Graphiti dual clocks + graph. FS Palace remains O(n) over tier files.

func (*PalaceStore) ListMemoryWithOptions 

func (ps *PalaceStore) ListMemoryWithOptions(opts ListMemoryOptions) []MemoryEntry

ListMemoryWithOptions returns entries ordered by event time with optional filters applied before Limit (K2 / s611 timeline surface; s1066 meta index when enabled).

Order of operations: collect candidates → session → time → tag filters → query substring → sort by entryEventTime → limit.

func (*PalaceStore) Load 

func (ps *PalaceStore) Load(id string, tier MemoryTier) (MemoryEntry, bool)

Load retrieves by ID and tier. Production: no side-effect prints, clean bool return.

func (*PalaceStore) MetaIndexLen 

func (ps *PalaceStore) MetaIndexLen() int

MetaIndexLen returns the number of entries currently cached in the meta index (0 if dirty/unbuilt). Intended for tests and observability — not a product API.

func (*PalaceStore) MetaIndexRebuilds 

func (ps *PalaceStore) MetaIndexRebuilds() uint64

MetaIndexRebuilds returns how many full tier-JSON walks rebuildMetaIndexLocked has run on this store. Tests use this to prove Write/unlink patched in place. Not a product API.

func (*PalaceStore) MultiHopRetrieve 

func (ps *PalaceStore) MultiHopRetrieve(opts MultiHopOptions) []MemoryEntry

MultiHopRetrieve performs multi-hop lite associative retrieval:

  1. Resolve seeds (SeedEntity and/or entities from SeedQuery search hits)
  2. ExpandRelatedEntitiesHops for each seed (min hop across seeds)
  3. Collect entries from default tiers matching any expanded entity
  4. Optional AsOf + SessionID filters BEFORE Limit
  5. Sort: lower min hop first (default), then event time desc within hop (legacy: seed-match first when PreferShorterHops is explicitly false)
  6. Limit

Default MaxHops=2 (clamped 1..4), Limit=20. Default tiers: Working + Contextual + Semantic (+ Archival if IncludeArchival). PreferShorterHops defaults true (path-aware ranking lite / s1067).

func (*PalaceStore) PerformCompaction 

func (ps *PalaceStore) PerformCompaction(
	targetTier MemoryTier,
	cfg CompactionConfig,
	generateFn func(prompt string) string,
	vectorCallback VectorStoreCallback,
) error

PerformCompaction runs agent-managed compaction with H-Mem temporal window + alpha constraints. Now respects LongMemEval fact protection and turn granularity. Product Write errors from SUMMARIZE / MERGE / CREATE_CORE_PRINCIPLE / ARCHIVE are returned.

func (*PalaceStore) PromoteToContextual 

func (ps *PalaceStore) PromoteToContextual(threshold float64)

PromoteToContextual promotes high-relevance Working entries (Phase 4.2)

func (*PalaceStore) ReadWithChainOfNote 

func (ps *PalaceStore) ReadWithChainOfNote(retrieved []MemoryEntry, query string) (string, error)

ReadWithChainOfNote formats retrieved MemoryEntry items into a Chain-of-Note style prompt.

func (*PalaceStore) SearchMemory 

func (ps *PalaceStore) SearchMemory(query string, tier *MemoryTier, limit int, vec []float32) []MemoryEntry

SearchMemory provides hybrid retrieval (keyword + vector + temporal) - Phase 4.1. Thin wrapper over SearchMemoryWithOptions with no session/time filters and ReRankTemporal=false.

func (*PalaceStore) SearchMemoryWithOptions 

func (ps *PalaceStore) SearchMemoryWithOptions(query string, opts SearchMemoryOptions) []MemoryEntry

SearchMemoryWithOptions provides hybrid retrieval with optional session, time-window, and temporal re-ranking. Uses the configured EmbeddingFunc for vector re-ranking.

func (*PalaceStore) SemanticRefine 

func (ps *PalaceStore) SemanticRefine(cluster []MemoryEntry) error

SemanticRefine (RecMem Phase 3) protects high-stake atomic facts from clusters. Products get SessionID / Timestamp from the parent and valid_from when unset.

func (*PalaceStore) SupersedeEntityFacts 

func (ps *PalaceStore) SupersedeEntityFacts(entityKey string, asOf time.Time) (int, error)

SupersedeEntityFacts finds entries matching entityKey (via EntryEntityKeys / entity: tags) that are still open at asOf (EntryValidAt), and writes valid_until = asOf (exclusive end matching existing FactsAsOf semantics). Does not delete entries.

Matching: normalize entityKey to lower-case trim; an entry matches when any key from EntryEntityKeys equals that normalized form (also lower-cased). Empty entityKey is a no-op (returns 0, nil).

Honesty: bi-temporal lite supersession — not automatic NLP contradiction detection, not full Zep dual-clock KG. Callers pass explicit entity keys.

Returns the count of updated entries.

func (*PalaceStore) Write 

func (ps *PalaceStore) Write(entry MemoryEntry) error

Write persists a MemoryEntry with an atomic write (temp file + rename).

Versioning is caller-managed and best-effort. Version==0 is stored as 1. archiveToVersions writes versions/memory-entries/<id>/v{Version}.json for the incoming entry; its errors are non-fatal and do not fail Write. A second Write of the same ID and Version overwrites both the live tier file and that snapshot. Callers who want history must increment Version themselves. This is not automatic overwrite versioning and not a hosted version store.

func (*PalaceStore) WriteAndSupersede 

func (ps *PalaceStore) WriteAndSupersede(entry MemoryEntry, supersedeKeys []string) error

WriteAndSupersede writes entry first (stamping valid_from=now when unset), then runs SupersedeEntityFacts for each supersedeKeys entry, excluding the newly written entry's ID so it is not closed by its own write.

asOf for supersession is the same UTC now used for valid_from stamping.

func (*PalaceStore) WriteLatent 

func (ps *PalaceStore) WriteLatent(entry MemoryEntry) error

WriteLatent stores entry in the subconscious latent buffer (RecMem Phase 1). No immediate promotion or LLM compaction is triggered.

Versioning is caller-managed and best-effort, same as Write: Version==0 is stored as 1; archiveToVersions errors are non-fatal; callers who want history must increment Version themselves. Overwrite does not auto-increment.

type SearchMemoryOptions 

type SearchMemoryOptions struct {
	// SessionID, when non-empty, keeps only entries with a matching SessionID.
	SessionID string
	// TimeFrom / TimeTo filter by entry event time (see entryEventTime).
	// Both bounds are inclusive when set.
	TimeFrom *time.Time
	TimeTo   *time.Time
	// AsOf, when non-nil, drops candidates where !EntryValidAt(e, *AsOf) before Limit
	// (K4 facts-as-of / validity window; see EntryValidAt).
	AsOf *time.Time
	// Limit caps the result set (default 10 when <= 0).
	Limit int
	// Tier, when non-nil, restricts candidates to that tier.
	Tier *MemoryTier
	// QueryVec, when non-empty, ranks candidates by cosine similarity.
	// Keyword hits (if the query has tokens of length >= 3) are kept ahead of
	// non-hits so hash embeddings cannot drop a literal match past Limit.
	QueryVec []float32
	// ReRankTemporal, when true, sorts results by CalculateRelevanceScore descending
	// after the keyword/vector path (before Limit). Keyword hits stay ahead of
	// non-hits so temporal re-rank cannot drop a literal match past Limit.
	ReRankTemporal bool
}

SearchMemoryOptions configures hybrid retrieval with optional session, time-window, as-of validity, and temporal re-ranking filters (s586 temporal retrieval; s616 AsOf).

type SearchResult 

type SearchResult struct {
	ID      string
	Score   float64
	Payload map[string]interface{}
}

SearchResult represents a single vector search hit with score and payload.

type VectorStore 

type VectorStore struct {
	Client     *qdrant.Client
	Collection string
	Enabled    bool
}

VectorStore provides optional vector capabilities using official Qdrant Go client v1 Production implementation with full dense/sparse query support, proper error handling, and payload extraction.

func NewVectorStore 

func NewVectorStore(rawURL, collection string) *VectorStore

NewVectorStore initializes the official Qdrant client.

func (*VectorStore) BatchUpsert 

func (vs *VectorStore) BatchUpsert(points []*qdrant.PointStruct) error

BatchUpsert upserts multiple points at once

func (*VectorStore) CreateBatchSparseCollections 

func (vs *VectorStore) CreateBatchSparseCollections(names []string) error

func (*VectorStore) CreateBatchSparseCollectionsContext 

func (vs *VectorStore) CreateBatchSparseCollectionsContext(ctx context.Context, names []string) error

func (*VectorStore) CreateBatchSparseCollectionsWithConcurrency 

func (vs *VectorStore) CreateBatchSparseCollectionsWithConcurrency(names []string, concurrency int) error

func (*VectorStore) CreateBatchSparseCollectionsWithConcurrencyContext 

func (vs *VectorStore) CreateBatchSparseCollectionsWithConcurrencyContext(ctx context.Context, names []string, concurrency int) error

CreateBatchSparseCollectionsWithConcurrencyContext is the context-aware version with custom concurrency.

func (*VectorStore) CreateBatchSparseCollectionsWithResults 

func (vs *VectorStore) CreateBatchSparseCollectionsWithResults(ctx context.Context, names []string) ([]CollectionCreateResult, error)

func (*VectorStore) CreateBatchSparseCollectionsWithResultsAndConcurrency 

func (vs *VectorStore) CreateBatchSparseCollectionsWithResultsAndConcurrency(ctx context.Context, names []string, concurrency int) ([]CollectionCreateResult, error)

func (*VectorStore) CreateCollection 

func (vs *VectorStore) CreateCollection(dim int) error

CreateCollection creates a dense vector collection with cosine distance.

func (*VectorStore) CreateSparseCollection 

func (vs *VectorStore) CreateSparseCollection() error

CreateSparseCollection creates a sparse vector collection.

func (*VectorStore) SearchByText 

func (vs *VectorStore) SearchByText(text string, limit int, filter map[string]interface{}, withPayload bool) ([]SearchResult, error)

SearchByText performs text-based semantic search.

func (*VectorStore) SearchSimilar 

func (vs *VectorStore) SearchSimilar(queryVec []float32, limit int, filter map[string]interface{}, withPayload bool) ([]SearchResult, error)

SearchSimilar performs dense vector similarity search using the official Qdrant Query API.

func (*VectorStore) SearchSparse 

func (vs *VectorStore) SearchSparse(queryIndices []uint32, queryValues []float32, limit int, filter map[string]interface{}, withPayload bool) ([]SearchResult, error)

SearchSparse performs sparse vector similarity search using the official Qdrant Query API.

func (*VectorStore) StoreSparseVector 

func (vs *VectorStore) StoreSparseVector(id string, indices []uint32, values []float32, payload map[string]interface{}) error

StoreSparseVector upserts a sparse vector with payload

func (*VectorStore) StoreVector 

func (vs *VectorStore) StoreVector(id string, vec []float32, payload map[string]interface{}) error

StoreVector upserts a dense vector with payload

type VectorStoreCallback 

type VectorStoreCallback func(id string, vec []float32, payload map[string]interface{}) error

VectorStoreCallback allows optional vector integration (e.g. Qdrant)

Source Files

View all Source files

Directories

Path Synopsis
cmd
longmemeval-bench command
longmemeval-server command
longmemeval-v2-bench command
internal
longmemeval
Download the default KnightsAnalytics bge-small-en-v1.5 ONNX model for local Palace recall.
 Download the default KnightsAnalytics bge-small-en-v1.5 ONNX model for local Palace recall.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.