memory

package
v0.0.0-...-f1b8c3c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 11, 2026 License: MIT Imports: 15 Imported by: 0

README

Vector Memory System

This package implements a vector-based memory system for goclaw with semantic search capabilities.

Architecture

Components
  1. Types (types.go) - Core data structures and interfaces
  2. Embeddings (embeddings.go) - Embedding provider implementations
  3. Store (store.go) - SQLite-based persistent storage
  4. Search (search.go) - High-level memory management API
  5. Vector (vector.go) - Vector math utilities

Features

  • Vector Embeddings: Support for OpenAI, Gemini, and extensible providers
  • Semantic Search: Cosine similarity search with configurable thresholds
  • Hybrid Search: Combine vector similarity with full-text search (FTS5)
  • Memory Types: Long-term, session, and daily notes
  • Caching: In-memory LRU cache for frequently accessed memories
  • Batch Operations: Efficient bulk embedding and storage

Usage

Basic Setup
import (
    "github.com/smallnest/goclaw/memory"
)

// Create embedding provider
provider, err := memory.NewOpenAIProvider(memory.DefaultOpenAIConfig(apiKey))
if err != nil {
    log.Fatal(err)
}

// Create store
store, err := memory.NewSQLiteStore(memory.DefaultStoreConfig(
    "/path/to/memory.db",
    provider,
))
if err != nil {
    log.Fatal(err)
}
defer store.Close()

// Create manager
manager, err := memory.NewMemoryManager(memory.DefaultManagerConfig(store, provider))
if err != nil {
    log.Fatal(err)
}
defer manager.Close()
Adding Memories
// Add a single memory
ve, err := manager.AddMemory(ctx, "User prefers dark mode",
    memory.MemorySourceLongTerm,
    memory.MemoryTypePreference,
    memory.MemoryMetadata{
        Tags: []string{"preference", "ui"},
        Importance: 0.8,
    })
if err != nil {
    log.Fatal(err)
}

// Add multiple memories
items := []memory.MemoryItem{
    {
        Text: "User works at TechCorp",
        Source: memory.MemorySourceLongTerm,
        Type: memory.MemoryTypeFact,
    },
    {
        Text: "Meeting scheduled for tomorrow",
        Source: memory.MemorySourceDaily,
        Type: memory.MemoryTypeContext,
    },
}
err = manager.AddMemoryBatch(ctx, items)
Searching
// Semantic search
results, err := manager.Search(ctx, "user preferences",
    memory.SearchOptions{
        Limit:    10,
        MinScore: 0.7,
        Hybrid:   true,
    })
if err != nil {
    log.Fatal(err)
}

for _, result := range results {
    fmt.Printf("Score: %.2f, Text: %s\n", result.Score, result.Text)
}

// Search by tag
preferenceMemories, err := manager.SearchByTag(ctx, "preference")

// Search by source
longTerm, err := manager.SearchBySource(ctx, memory.MemorySourceLongTerm)

Configuration

Store Options
config := memory.StoreConfig{
    DBPath:              "/path/to/memory.db",
    Provider:            provider,
    EnableVectorSearch:  true,   // Requires sqlite-vec
    EnableFTS:           true,   // Requires FTS5
    VectorExtensionPath: "",     // Auto-detect if empty
}
Search Options
opts := memory.SearchOptions{
    Limit:        10,              // Max results
    MinScore:     0.7,             // Minimum similarity (0-1)
    Sources:      []MemorySource{MemorySourceLongTerm},
    Types:        []MemoryType{MemoryTypeFact},
    Hybrid:       true,            // Enable hybrid search
    VectorWeight: 0.7,             // Weight for vector similarity
    TextWeight:   0.3,             // Weight for keyword match
}

Dependencies

  • github.com/glebarez/sqlite - Pure Go SQLite driver
  • github.com/google/uuid - UUID generation

Optional Extensions

  • sqlite-vec: Vector similarity search extension

  • FTS5: Full-text search (built into SQLite)

Integration with Agent Context

To integrate with the agent context builder:

import "github.com/smallnest/goclaw/memory"

// In agent/context.go
type ContextBuilder struct {
    // ... existing fields
    memoryManager *memory.MemoryManager
}

func (b *ContextBuilder) buildSystemPromptWithSkills(...) string {
    // ... existing code

    // Add relevant memories
    if b.memoryManager != nil {
        results, err := b.memoryManager.Search(ctx,
            "user preferences and context",
            memory.SearchOptions{
                Limit:    5,
                MinScore: 0.7,
            })
        if err == nil && len(results) > 0 {
            parts = append(parts, "## Relevant Memories\n\n")
            for _, r := range results {
                parts = append(parts, fmt.Sprintf("- %s\n", r.Text))
            }
        }
    }

    return fmt.Sprintf("%s\n\n", joinNonEmpty(parts, "\n\n---\n\n"))
}

Performance Considerations

  • Batch Size: OpenAI supports up to 2048 texts per batch
  • Caching: Enable for frequently accessed memories
  • Vector Extension: Optional but recommended for large datasets
  • FTS5: Faster than substring matching for text search

Future Enhancements

  1. Additional embedding providers (Voyage, local models)
  2. Automatic memory file indexing
  3. Session transcript integration
  4. Memory importance scoring
  5. Automatic memory pruning
  6. Hierarchical memory organization

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Add 

func Add(a, b []float32) ([]float32, error)

Add adds two vectors

func ChunkText 

func ChunkText(text string, maxTokens int) []string

ChunkText splits text into chunks suitable for embedding

func ComputeHash 

func ComputeHash(vec []float32) uint64

ComputeHash computes a simple hash of a vector for caching purposes

func CosineSimilarity 

func CosineSimilarity(a, b []float32) (float64, error)

CosineSimilarity computes the cosine similarity between two vectors

func DotProduct 

func DotProduct(a, b []float32) (float64, error)

DotProduct computes the dot product of two vectors

func EuclideanDistance 

func EuclideanDistance(a, b []float32) (float64, error)

EuclideanDistance computes the Euclidean distance between two vectors

func Magnitude 

func Magnitude(vec []float32) (float64, error)

Magnitude computes the magnitude (length) of a vector

func Mean 

func Mean(vectors [][]float32) ([]float32, error)

Mean computes the mean vector of multiple vectors

func Multiply 

func Multiply(vec []float32, scalar float64) ([]float32, error)

Multiply multiplies a vector by a scalar

func Normalize 

func Normalize(vec []float32) ([]float32, error)

Normalize normalizes a vector to unit length

func Subtract 

func Subtract(a, b []float32) ([]float32, error)

Subtract subtracts vector b from vector a

Types

type EmbeddingProvider 

type EmbeddingProvider interface {
	// Embed generates a single embedding
	Embed(text string) ([]float32, error)
	// EmbedBatch generates multiple embeddings in one call
	EmbedBatch(texts []string) ([][]float32, error)
	// Dimension returns the dimension of embeddings
	Dimension() int
	// MaxBatchSize returns the maximum batch size
	MaxBatchSize() int
}

EmbeddingProvider defines the interface for generating embeddings

type GeminiProvider 

type GeminiProvider struct {
	// contains filtered or unexported fields
}

GeminiProvider implements EmbeddingProvider using Google's Gemini API This is a placeholder for future implementation

func NewGeminiProvider 

func NewGeminiProvider(apiKey, model string) (*GeminiProvider, error)

NewGeminiProvider creates a new Gemini embedding provider

func (*GeminiProvider) Dimension 

func (p *GeminiProvider) Dimension() int

Dimension returns the dimension of embeddings

func (*GeminiProvider) Embed 

func (p *GeminiProvider) Embed(text string) ([]float32, error)

Embed generates a single embedding

func (*GeminiProvider) EmbedBatch 

func (p *GeminiProvider) EmbedBatch(texts []string) ([][]float32, error)

EmbedBatch generates multiple embeddings in one call

func (*GeminiProvider) MaxBatchSize 

func (p *GeminiProvider) MaxBatchSize() int

MaxBatchSize returns the maximum batch size

type ManagerConfig 

type ManagerConfig struct {
	Store        Store
	Provider     EmbeddingProvider
	CacheMaxSize int
}

ManagerConfig configures the memory manager

func DefaultManagerConfig 

func DefaultManagerConfig(store Store, provider EmbeddingProvider) ManagerConfig

DefaultManagerConfig returns default manager configuration

type MemoryItem 

type MemoryItem struct {
	Text     string
	Source   MemorySource
	Type     MemoryType
	Metadata MemoryMetadata
}

MemoryItem represents a memory to be added

type MemoryManager 

type MemoryManager struct {
	// contains filtered or unexported fields
}

MemoryManager manages memory storage and retrieval

func NewMemoryManager 

func NewMemoryManager(config ManagerConfig) (*MemoryManager, error)

NewMemoryManager creates a new memory manager

func (*MemoryManager) AddMemory 

func (m *MemoryManager) AddMemory(ctx context.Context, text string, source MemorySource, memType MemoryType, metadata MemoryMetadata) (*VectorEmbedding, error)

AddMemory adds a new memory with automatic embedding generation

func (*MemoryManager) AddMemoryBatch 

func (m *MemoryManager) AddMemoryBatch(ctx context.Context, items []MemoryItem) error

AddMemoryBatch adds multiple memories with automatic embedding generation

func (*MemoryManager) ClearCache 

func (m *MemoryManager) ClearCache()

ClearCache clears the in-memory cache

func (*MemoryManager) Close 

func (m *MemoryManager) Close() error

Close closes the memory manager

func (*MemoryManager) Delete 

func (m *MemoryManager) Delete(ctx context.Context, id string) error

Delete removes a memory

func (*MemoryManager) Get 

func (m *MemoryManager) Get(ctx context.Context, id string) (*VectorEmbedding, error)

Get retrieves a memory by ID

func (*MemoryManager) GetStats 

func (m *MemoryManager) GetStats(ctx context.Context) (*MemoryStats, error)

GetStats returns statistics about the memory store

func (*MemoryManager) List 

func (m *MemoryManager) List(ctx context.Context, filter func(*VectorEmbedding) bool) ([]*VectorEmbedding, error)

List lists all memories with optional filtering

func (*MemoryManager) Search 

func (m *MemoryManager) Search(ctx context.Context, query string, opts SearchOptions) ([]*SearchResult, error)

Search searches for similar memories

func (*MemoryManager) SearchBySource 

func (m *MemoryManager) SearchBySource(ctx context.Context, source MemorySource) ([]*VectorEmbedding, error)

SearchBySource searches memories by source

func (*MemoryManager) SearchByTag 

func (m *MemoryManager) SearchByTag(ctx context.Context, tag string) ([]*VectorEmbedding, error)

SearchByTag searches memories by tag

func (*MemoryManager) SearchByText 

func (m *MemoryManager) SearchByText(ctx context.Context, query string) ([]*VectorEmbedding, error)

SearchByText searches memories by text content (simple substring match)

func (*MemoryManager) SearchByType 

func (m *MemoryManager) SearchByType(ctx context.Context, memType MemoryType) ([]*VectorEmbedding, error)

SearchByType searches memories by type

func (*MemoryManager) Update 

func (m *MemoryManager) Update(ctx context.Context, ve *VectorEmbedding) error

Update updates an existing memory

type MemoryMetadata 

type MemoryMetadata struct {
	// FilePath is the source file path (if from file)
	FilePath string `json:"file_path,omitempty"`
	// LineNumber is the line in the source file
	LineNumber int `json:"line_number,omitempty"`
	// SessionKey is the session identifier (if from session)
	SessionKey string `json:"session_key,omitempty"`
	// Tags are user-defined tags
	Tags []string `json:"tags,omitempty"`
	// Importance is a user-assigned importance score (0-1)
	Importance float64 `json:"importance,omitempty"`
	// AccessCount tracks how often this memory was accessed
	AccessCount int `json:"access_count,omitempty"`
	// LastAccessed is when this memory was last retrieved
	LastAccessed time.Time `json:"last_accessed,omitempty"`
}

MemoryMetadata contains additional information about a memory

type MemorySource 

type MemorySource string

MemorySource represents where a memory entry originates 

const (
	// MemorySourceLongTerm is from MEMORY.md and related files
	MemorySourceLongTerm MemorySource = "longterm"
	// MemorySourceSession is from conversation history
	MemorySourceSession MemorySource = "session"
	// MemorySourceDaily is from daily notes (YYYY-MM-DD.md)
	MemorySourceDaily MemorySource = "daily"
)

type MemoryStats 

type MemoryStats struct {
	TotalCount   int                  `json:"total_count"`
	SourceCounts map[MemorySource]int `json:"source_counts"`
	TypeCounts   map[MemoryType]int   `json:"type_counts"`
	CacheSize    int                  `json:"cache_size"`
	CacheMaxSize int                  `json:"cache_max_size"`
}

MemoryStats contains statistics about the memory store

type MemoryType 

type MemoryType string

MemoryType represents the type of memory content 

const (
	// MemoryTypeFact is a factual piece of information
	MemoryTypeFact MemoryType = "fact"
	// MemoryTypePreference is user preference or setting
	MemoryTypePreference MemoryType = "preference"
	// MemoryTypeContext is situational context
	MemoryTypeContext MemoryType = "context"
	// MemoryTypeConversation is conversation summary
	MemoryTypeConversation MemoryType = "conversation"
)

type OpenAIConfig 

type OpenAIConfig struct {
	APIKey     string
	BaseURL    string
	Model      string
	Timeout    time.Duration
	MaxRetries int
}

OpenAIConfig configures the OpenAI embedding provider

func DefaultOpenAIConfig 

func DefaultOpenAIConfig(apiKey string) OpenAIConfig

DefaultOpenAIConfig returns default OpenAI configuration

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

OpenAIProvider implements EmbeddingProvider using OpenAI's API

func NewOpenAIProvider 

func NewOpenAIProvider(config OpenAIConfig) (*OpenAIProvider, error)

NewOpenAIProvider creates a new OpenAI embedding provider

func (*OpenAIProvider) Dimension 

func (p *OpenAIProvider) Dimension() int

Dimension returns the dimension of embeddings

func (*OpenAIProvider) Embed 

func (p *OpenAIProvider) Embed(text string) ([]float32, error)

Embed generates a single embedding

func (*OpenAIProvider) EmbedBatch 

func (p *OpenAIProvider) EmbedBatch(texts []string) ([][]float32, error)

EmbedBatch generates multiple embeddings in one call

func (*OpenAIProvider) MaxBatchSize 

func (p *OpenAIProvider) MaxBatchSize() int

MaxBatchSize returns the maximum batch size

type SQLiteStore 

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore implements the Store interface using SQLite

func NewSQLiteStore 

func NewSQLiteStore(config StoreConfig) (*SQLiteStore, error)

NewSQLiteStore creates a new SQLite-based memory store

func (*SQLiteStore) Add 

func (s *SQLiteStore) Add(embedding *VectorEmbedding) error

Add adds a memory to the store

func (*SQLiteStore) AddBatch 

func (s *SQLiteStore) AddBatch(embeddings []*VectorEmbedding) error

AddBatch adds multiple memories in one transaction

func (*SQLiteStore) Close 

func (s *SQLiteStore) Close() error

Close closes the store

func (*SQLiteStore) Delete 

func (s *SQLiteStore) Delete(id string) error

Delete removes a memory by ID

func (*SQLiteStore) Get 

func (s *SQLiteStore) Get(id string) (*VectorEmbedding, error)

Get retrieves a memory by ID

func (*SQLiteStore) List 

func (s *SQLiteStore) List(filter func(*VectorEmbedding) bool) ([]*VectorEmbedding, error)

List lists all memories with optional filtering

func (*SQLiteStore) Search 

func (s *SQLiteStore) Search(query []float32, opts SearchOptions) ([]*SearchResult, error)

Search performs vector similarity search

func (*SQLiteStore) Update 

func (s *SQLiteStore) Update(embedding *VectorEmbedding) error

Update updates an existing memory

type SearchOptions 

type SearchOptions struct {
	// Limit is the maximum number of results to return
	Limit int `json:"limit"`
	// MinScore is the minimum similarity score (0-1)
	MinScore float64 `json:"min_score"`
	// Sources filters which memory sources to search
	Sources []MemorySource `json:"sources,omitempty"`
	// Types filters which memory types to search
	Types []MemoryType `json:"types,omitempty"`
	// Hybrid enables hybrid vector+keyword search
	Hybrid bool `json:"hybrid"`
	// VectorWeight is the weight for vector similarity in hybrid search (0-1)
	VectorWeight float64 `json:"vector_weight"`
	// TextWeight is the weight for keyword match in hybrid search (0-1)
	TextWeight float64 `json:"text_weight"`
}

SearchOptions configures memory search behavior

func DefaultSearchOptions 

func DefaultSearchOptions() SearchOptions

DefaultSearchOptions returns sensible default search options

type SearchResult 

type SearchResult struct {
	VectorEmbedding
	Score       float64 `json:"score"`
	MatchedText string  `json:"matched_text"`
	Highlight   string  `json:"highlight,omitempty"`
}

SearchResult represents a memory search result with relevance score

type Store 

type Store interface {
	// Add adds a memory to the store
	Add(embedding *VectorEmbedding) error
	// AddBatch adds multiple memories in one transaction
	AddBatch(embeddings []*VectorEmbedding) error
	// Search performs similarity search
	Search(query []float32, opts SearchOptions) ([]*SearchResult, error)
	// Get retrieves a memory by ID
	Get(id string) (*VectorEmbedding, error)
	// Delete removes a memory by ID
	Delete(id string) error
	// Update updates an existing memory
	Update(embedding *VectorEmbedding) error
	// List lists all memories with optional filtering
	List(filter func(*VectorEmbedding) bool) ([]*VectorEmbedding, error)
	// Close closes the store
	Close() error
}

Store defines the interface for memory storage

type StoreConfig 

type StoreConfig struct {
	// DBPath is the path to the SQLite database file
	DBPath string
	// Provider is the embedding provider to use
	Provider EmbeddingProvider
	// EnableVectorSearch enables vector similarity search (requires sqlite-vec)
	EnableVectorSearch bool
	// EnableFTS enables full-text search
	EnableFTS bool
	// VectorExtensionPath is the path to the sqlite-vec extension
	VectorExtensionPath string
}

StoreConfig configures the SQLite memory store

func DefaultStoreConfig 

func DefaultStoreConfig(dbPath string, provider EmbeddingProvider) StoreConfig

DefaultStoreConfig returns default store configuration

type VectorEmbedding 

type VectorEmbedding struct {
	ID        string         `json:"id"`
	Vector    []float32      `json:"vector"`
	Dimension int            `json:"dimension"`
	Text      string         `json:"text"`
	Source    MemorySource   `json:"source"`
	Type      MemoryType     `json:"type"`
	CreatedAt time.Time      `json:"created_at"`
	UpdatedAt time.Time      `json:"updated_at"`
	Metadata  MemoryMetadata `json:"metadata"`
}

VectorEmbedding represents a vector embedding with metadata

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.