hippocampus

README

Hippocampus

A finite, biological-inspired memory storage engine for log retention, audit trails, and context management.

---

💡 Why Hippocampus?

Traditional storage engines rely on Time-To-Live (TTL) or fixed FIFO queues to manage bounded disk space. But age alone is a poor indicator of value: critical system anomalies, high-impact audit events, and frequently referenced context often get purged simply because they crossed an arbitrary time threshold.

Hippocampus applies principles from human memory consolidation to solve long-term data retention under finite capacity. Rather than indiscriminately truncating or expiring data, it continuously evaluates significance, access frequency, and relationships—retaining the highest-value context while gracefully degrading low-value noise.

• Relative Significance & Ranking: Insert events dynamically relative to adjacent records (ABOVE, BELOW, or BETWEEN) without enforcing rigid, static importance scales.
• Reinforcement through Recall: Accessing or querying a record strengthens its retention weight, protecting high-demand operational data from decay.
• Sleep & Consolidation: Runs periodic background consolidation cycles to apply decay models, compact space, and distill clusters of episodic details into compact semantic summaries.
• Durable & Compliance-Safe: Embedded or centralized deployment backed by SQLite (WAL mode), PostgreSQL, or MySQL. Includes configurable minimum retention floors to guarantee compliance windows regardless of storage pressure.

---

⚡ 30-Second Quick Start

Try Hippocampus locally with zero external dependencies (uses pure-Go embedded SQLite):

1. Run the Demo Stack

git clone https://github.com/fastbean-au/hippocampus.git
cd hippocampus
./demo/run.sh

2. Access the UI & Services

• Embedded Web Console: Open http://localhost:8080/ui to browse, search, and observe memory consolidation in real time.
• gRPC Endpoint: Listening on localhost:50051
• HTTP Gateway: Listening on localhost:8080
• LGTM stack: Listening on http://localhost:3000 to view live metrics in Grafana.

---

🚀 Docker Setup

Run Hippocampus in containerized environments with pre-configured compose files:

# Embedded SQLite (Stateless binary, volume-backed DB)
docker compose up --build

# PostgreSQL Backed
docker compose -f docker/docker-compose.postgres.yaml up --build

# Centralized Setup (PostgreSQL + OpenSearch Content Indexing)
docker compose -f docker/docker-compose.corporate.yaml up --build

---

🏗️ Deployment Topology & Scaling

Hippocampus scales cleanly using two primary deployment patterns depending on store ownership:

[ Isolated Multi-Tenant / Embedded ]        [ High-Throughput Centralized ]

+----------------------+                  +-------------------------+
|  Tenant A / Device   |                  |   Consolidating Node    |
| (1 Instance = 1 DB)  |                  |  (Runs Sleep/Eviction)  |
+----------------------+                  +-----------+-------------+
                                                      |
+----------------------+                 +------------+--------------+
|  Tenant B / Device   |                 | Shared DB (Postgres/MySQL)|
| (1 Instance = 1 DB)  |                 +------------+--------------+
+----------------------+                              |
                                         +------------+--------------+
                                         |   Read / Write Replicas   |
                                         | (consolidation.enabled=f) |
                                         +---------------------------+

1. One Instance per Store (Recommended): Run independent, lightweight Hippocampus instances per subsystem, client tenant, or edge node using SQLite or dedicated databases.
2. Shared Store with Replicas: Scale centralized stores by running one consolidating instance (consolidation.enabled: true) alongside any number of stateless read/write HTTP/gRPC replicas (consolidation.enabled: false).

---

📚 Documentation Index

Detailed operational and architectural guides live under docs/:

Guide	Description
🎬 Getting Started	Step-by-step build, initial config, and first gRPC/HTTP requests.
⚙️ Configurability	Exhaustive key reference for TLS, auth, storage drivers, and listeners.
🧠 Memory Consolidation	Deep dive on decay algorithms, capacity targets, and summarization.
🛠️ Operations & Deployment	Sizing storage, PostgreSQL/MySQL tuning, backups, and security hardening.
📊 Performance Benchmarks	Throughput sweeps across SQLite, Postgres, and MySQL under heavy loads.
📐 Use Cases & Patterns	Embedded vs. centralized topologies and data transfer strategies.
🧪 Demonstrations	Worked scenarios using real-world data shapes and data generators.

---

🔒 Security & Hardening

Hippocampus is production-hardened out of the box:

• Built-in Authentication: JWT bearer tokens with mandatory expiration (exp) and zero-downtime rotation via auth.signingKeys.
• Transport Security: Pinned TLS 1.2+ floor for both internal and external communication.
• Storage Isolation: Driver error masking behind standard gRPC status codes to prevent database schema leaks.
• Client Isolation: Per-client request attribution, execution query timeouts, and stream concurrency limits.

Read the Security Section in Operations for details on proxying behind sidecars, token revocation files, and network boundaries.

---

⚠️ Key Limitations

• Single Consolidator Rule: Only one instance may perform consolidation/decay tasks per store to prevent race conditions during database compaction.
• Opaque Payloads: Memory payloads are stored as raw bytes; summaries must be constructed upstream by client applications and submitted via ReplaceMemoriesWithSummary.
• Eventually Consistent Search: The OpenSearch index is secondary and asynchronous. Primary database reads remain strictly consistent, while background sweeps handle reconciliation for content search.

---

📄 License

Distributed under the terms specified in the repository. See LICENSE for details.