knirvbase

README

KNIRVBASE (Go)

✅ Overview

KNIRVBASE is a lightweight, local-first streaming database prototype implemented in Go. It demonstrates a minimal, self-contained implementation of:

• PQC Encryption Layer: Post-quantum cryptography using Kyber-768 (encryption) and Dilithium-3 (signatures) for secure data storage
• CRDT-based conflict resolution using vector clocks,
• Local file-backed storage for metadata and blobs (blobs are stored locally),
• Real P2P networking with DHT-like peer discovery and TCP-based communication,
• A small, human-friendly query language KNIRVQL for simple GET/SET/DELETE and basic vector similarity queries.

This package is intended as an architectural prototype and reference implementation for streaming collections and synchronization logic used across the KNIRV ecosystem.

---

🔍 Features

• PQC Encryption at Rest: Field-level encryption for sensitive data using Kyber-768 KEM + AES-256-GCM, with Dilithium-3 signatures for integrity
• Local-first operations and background sync
• CRDT resolve using vector clocks (merge rules + LWW tie-breakers)
• File-based storage for documents and local blob files
• Real P2P networking with TCP connections and DHT-like peer discovery
• KNIRVQL parser and executor for quick interactive operations
• Convenience command-line demo in cmd/main.go

---

🚀 Quickstart

Prerequisites

• Go 1.24 (see go.mod) — ensure your toolchain matches or is compatible: go version

Build

make build
# or
go build -o ./bin/knirvbase ./cmd

Run

The binary stores data in your OS application data directory by default. On Linux it uses $XDG_DATA_HOME or falls back to ~/.local/share/knirvbase.

./bin/knirvbase

The included cmd/main.go file demonstrates:

• Creating a streaming database instance
• Creating/joining a network
• Creating two collections: auth and memory
• Example KNIRVQL usage to SET and GET authentication keys
• Inserting a MEMORY entry with a vector payload and querying via similarity

Press Ctrl+C to exit the running demo.

---

📁 Storage Layout

• Data directory (default): $XDG_DATA_HOME or ~/.local/share/knirvbase
• Per-collection JSON files: <datadir>/<collection>/<id>.json
• Blobs (for MEMORY entries) are saved under <datadir>/<collection>/blobs/<id> and are not automatically synchronized across peers — only metadata and vectors are synchronized.

Why blobs are not synced: to preserve network bandwidth and storage efficiency. The system synchronizes discovery metadata (including blob references) rather than raw blobs.

---

🧭 KNIRVQL (Query Language) — Examples

• Set an auth key:

SET google_maps_api_key = "AIzaSy..."

• Get an auth key:

GET AUTH WHERE key = "google_maps_api_key"

• Insert a memory item (demonstrated programmatically in cmd/main.go)

• Get similar memory entries (vector search):

GET MEMORY WHERE source = "web-scrape" SIMILAR TO [0.45, 0.12] LIMIT 10

The language is intentionally minimal and aimed at quick integration and demos — for production you will likely expose richer query primitives or integrate with a dedicated vector index.

---

📦 Package Overview (what's inside)

• cmd/ — small demo CLI (cmd/main.go) showing initialization and sample operations.
• internal/crypto/pqc — Post-quantum cryptography: Kyber-768 encryption, Dilithium-3 signatures, key management.
• internal/database — StreamingDatabase: high-level database orchestration and collection factory.
• internal/collection — DistributedCollection + LocalCollection: local storage, CRDT operation emission, sync logic.
• internal/storage — FileStorage: file-based persistence and blob handling with PQC encryption.
• internal/network — Network interface + NetworkManager: real P2P networking with TCP connections and DHT-like peer discovery.
• internal/resolver — CRDT resolver logic and helpers for converting to/from streaming documents.
• internal/clock — vector clock implementation and comparison utilities.
• internal/query — KNIRVQL parser and query execution.
• internal/types — core types (Document, CRDTOperation, NetworkConfig, ProtocolMessage, etc.)

---

🛠 Development & Testing

• Run tests (where present):

go test ./...

• Run performance benchmarks:

make bench
# or
go test -bench=. -benchmem ./internal/benchmarks

• Run SLA validation tests:

make bench-sla
# or
go test -run=TestBenchmarkSLAs -v ./internal/benchmarks

• Format & vet:

gofmt -w .
go vet ./...

• Manage modules:

go mod tidy

Note: The codebase includes a full P2P networking implementation with TCP connections and DHT-like peer discovery for streaming operation.

---

📊 Performance Benchmarks

KNIRVBASE includes a comprehensive benchmark suite that validates performance against ASIC-Shield SLA requirements:

SLA Targets (ASIC-Shield Integration)

• Credential Insert: p99 < 10ms
• Credential Query: p99 < 5ms
• Authentication Workflow: p99 < 500ms (including 100M KDF iterations)
• PQC Encryption: < 20ms per operation
• Large Scale: No performance degradation with 10K+ credentials

Running Benchmarks

# Run all benchmarks
make bench-all

# Run SLA validation
make bench-sla

# Generate CPU/memory profiles
make bench-profile

Benchmark Results

The benchmark suite (benchmarks_test.go) includes:

• BenchmarkCredentialInsert: Measures credential storage performance
• BenchmarkCredentialQuery: Measures credential lookup performance
• BenchmarkPQCCrypto: Measures PQC encryption/decryption overhead
• BenchmarkAuthWorkflow: Simulates full authentication workflow
• BenchmarkLargeScale: Tests performance with 10K credentials

Integration tests (benchmark_integration_test.go) validate that results meet SLA targets and detect performance regressions.

---

⚠️ Limitations & Security Notes

• PQC Encryption at Rest: Field-level encryption for all sensitive data across collections (credentials, pqc_keys, sessions, audit_log, threat_events, access_control). Encrypts specific fields like hash, salt, token_hash, details, indicators, permissions, etc. Uses Kyber-768 + AES-256-GCM for confidentiality and Dilithium-3 for integrity. Master key must be configured for encryption to be active.
• Data Streaming: Real TCP and socket-based P2P streaming with DHT-like peer discovery enables true distributed operation across multiple nodes.
• Blob handling: Blobs are stored locally and only referenced in synchronized metadata; no blob distribution is implemented here.
• No authentication for network messages: This prototype does not implement cryptographic signing or authenticated transports — real deployments must use TLS/Noise and authenticated peer identities.

---

📚 Design & Reference

The repository includes Distributed_Database_Implementation_go.md which documents architecture, rationale, and design decisions in depth — consult it for more detail on synchronization heuristics, CRDT rules, and future extensions (IPFS integration, libp2p transport, vector indexes).

---

💡 Contributing

• Open an issue for feature requests or bug reports
• Create a PR with tests and descriptions of changes
• Keep code and docs consistent with Go idioms and the repository's architecture

---

📜 License

See the repository LICENSE.

---

Suggested next steps:

• Add usage examples as runnable scripts
• Add unit tests for untested modules (network, resolver)
• Add performance benchmarks for P2P operations