qdb

package module
v0.0.0-...-24fdf7d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 7, 2025 License: MIT Imports: 11 Imported by: 0

README

Go Reference Go Report Card

QDB is a comprehensive database library that combines Redis-style key-value operations with advanced vector search capabilities, all implemented in pure Go without external dependencies.

Features

Feature Algorithm/Structure
Key-Value Search Redis-style O(1) hash table
Vector Search (ANN) HNSW (Hierarchical Navigable Small World)
Exact Vector Search K-Nearest Neighbors (K-NN)
Combined Search Hybrid query with pre/post-filtering
Key Features
  • Zero External Dependencies: Pure Go implementation
  • Redis-like Performance: O(1) hash table operations with bucket chaining
  • Vector Search: Both approximate (HNSW) and exact (K-NN) search
  • Flexible Storage: File-based or in-memory storage
  • Type Safety: Support for various data types (string, int, float64, bool, bytes, JSON)
  • Concurrent Safe: Thread-safe operations with fine-grained locking
  • Filtering: Advanced filtering capabilities for vector search

Installation

go get github.com/mew-sh/qdb

Quick Start

package main

import (
    "context"
    "fmt"
    "log"
    
    "github.com/mew-sh/qdb"
)

func main() {
    ctx := context.Background()
    
    // Create database instance
    config := qdb.DefaultConfig()
    db, err := qdb.New(config)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()
    
    // Basic key-value operations
    err = db.Set(ctx, "key", "value")
    if err != nil {
        log.Fatal(err)
    }
    
    val, err := db.Get(ctx, "key").Result()
    if err != nil {
        log.Fatal(err)
    }
    
    fmt.Printf("Value: %v\n", val)
}

Usage Examples

Basic Key-Value Operations
ctx := context.Background()

// Set different data types
db.Set(ctx, "name", "Alice")
db.Set(ctx, "age", 30)
db.Set(ctx, "score", 95.5)
db.Set(ctx, "active", true)

// Get values with type conversion
name, err := db.Get(ctx, "name").String()
age, err := db.Get(ctx, "age").Int()
score, err := db.Get(ctx, "score").Float()

// Check existence
exists, err := db.Exists(ctx, "name")

// Delete
err = db.Delete(ctx, "name")
Vector Operations
// Store vectors with metadata
vector := []float64{1.0, 2.0, 3.0, 4.0}
metadata := map[string]interface{}{
    "category": "A",
    "priority": 1,
}

err := db.SetVector(ctx, "vec1", vector, metadata)

// Approximate search using HNSW
query := []float64{1.1, 2.1, 3.1, 4.1}
results, err := db.SearchVector(ctx, query, 5) // top 5 results

// Exact search using K-NN
exactResults, err := db.SearchVectorExact(ctx, query, 5)

// Search with filters
filter := &qdb.FilterCondition{
    Field:    "category",
    Operator: "eq",
    Value:    "A",
}
filteredResults, err := db.SearchVectorExact(ctx, query, 5, filter)
Configuration
config := &qdb.Config{
    DatabasePath: "mydb.db",    // File path or ":memory:" for in-memory
    UseMemory:    false,        // Use in-memory storage
    HNSWConfig: &qdb.HNSWConfig{
        M:              16,      // Connections per node
        EfConstruction: 200,     // Search width during construction
        EfSearch:       50,      // Search width during query
        MaxLayers:      16,      // Maximum layers in the graph
        Ml:             1.0 / math.Log(2.0), // Level generation parameter
    },
}

db, err := qdb.New(config)

Architecture

Hash Table Implementation

QDB uses a Redis-style hash table with the following characteristics:

  • FNV-1a Hash Function: Fast and well-distributed hashing
  • Bucket Chaining: Collision resolution using linked lists
  • Dynamic Resizing: Automatic resizing when load factor exceeds threshold
  • Concurrent Access: Read-write locks for thread safety
HNSW (Hierarchical Navigable Small World)
  • Multi-layer Graph: Hierarchical structure for efficient search
  • Skip List Inspired: Higher layers for long-range connections
  • Logarithmic Complexity: O(log N) search time complexity
  • High Recall: Excellent approximate search quality
  • Brute Force: Examines all vectors for perfect accuracy
  • Distance Metrics: Euclidean distance calculation
  • Filtering Support: Pre and post-filtering capabilities
Storage Engine
File Storage
  • Binary Format: Efficient binary serialization
  • Length Prefixed: Self-describing record format
  • Append-Only: Fast write operations
  • Compaction: Garbage collection for deleted records
Memory Storage
  • Hash Map: In-memory key-value storage
  • Zero Persistence: Data lost on restart
  • Maximum Performance: No I/O overhead

Performance

Hash Table Performance
  • Set Operations: ~1M ops/sec
  • Get Operations: ~2M ops/sec
  • Memory Usage: ~40 bytes per key-value pair overhead
Vector Search Performance
  • HNSW Construction: O(log N) per insertion
  • HNSW Search: O(log N) per query
  • Exact Search: O(N) per query
  • Memory Usage: ~100 bytes per vector overhead

Filter Operators

Operator Description Example
eq Equal to category = "A"
ne Not equal to status != "inactive"
gt Greater than priority > 5
lt Less than score < 100
gte Greater than or equal rating >= 4.0
lte Less than or equal age <= 65
in In array category in ["A", "B"]

Data Types

QDB supports the following data types:

  • string: Text data
  • int: Integer numbers
  • float64: Floating point numbers
  • bool: Boolean values
  • []byte: Binary data
  • slices: Any slice type ([]int, []string, []float64, etc.) via JSON serialization
  • maps: Any map type via JSON serialization
  • structs: Custom types via JSON serialization
  • JSON: Complex objects (maps, slices, etc.)
Slice Support

QDB fully supports slice data types through automatic JSON serialization:

// Store different slice types
intSlice := []int{1, 2, 3, 4, 5}
stringSlice := []string{"a", "b", "c"}
floatSlice := []float64{1.1, 2.2, 3.3}

db.Set(ctx, "ints", intSlice)
db.Set(ctx, "strings", stringSlice) 
db.Set(ctx, "floats", floatSlice)

// Retrieve slices
// In memory: preserves original type
intResult, _ := db.Get(ctx, "ints").Result() // []int

// From storage: becomes []interface{} due to JSON
// But the data is preserved and can be converted back
if slice, ok := intResult.([]interface{}); ok {
    for _, v := range slice {
        intVal := int(v.(float64)) // JSON numbers are float64
    }
}

Thread Safety

QDB is designed to be thread-safe:

  • Hash Table: Uses read-write mutexes for concurrent access
  • Vector Index: Protected by mutexes during updates
  • Storage: Synchronized file operations
  • HNSW: Thread-safe graph construction and search

Limitations

  • Memory Usage: All data must fit in memory
  • Vector Dimensions: All vectors must have the same dimension
  • File Format: Simple format, not optimized for very large datasets
  • Transactions: No transaction support yet

Examples

See the example directory for comprehensive usage examples.

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Documentation

Overview

Package qdb examples and documentation.

Basic Usage

Here's a simple example of using QDB for key-value operations: 

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/mew-sh/qdb"
)

func main() {
    ctx := context.Background()

    // Create database with default config
    db, err := qdb.New(qdb.DefaultConfig())
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // Store values
    db.Set(ctx, "user:1:name", "Alice")
    db.Set(ctx, "user:1:age", 30)
    db.Set(ctx, "user:1:score", 95.5)

    // Retrieve values
    name, _ := db.Get(ctx, "user:1:name").String()
    age, _ := db.Get(ctx, "user:1:age").Int()
    score, _ := db.Get(ctx, "user:1:score").Float()

    fmt.Printf("User: %s, Age: %d, Score: %.1f\n", name, age, score)
}

QDB supports efficient vector similarity search: 

// Store vectors with metadata
vector1 := []float64{1.0, 2.0, 3.0, 4.0}
metadata1 := map[string]interface{}{
    "category": "document",
    "tags": []string{"important", "review"},
}
db.SetVector(ctx, "doc1", vector1, metadata1)

// Search for similar vectors
query := []float64{1.1, 2.1, 3.1, 4.1}
results, err := db.SearchVector(ctx, query, 5)
if err != nil {
    log.Fatal(err)
}

for _, result := range results {
    fmt.Printf("Key: %s, Score: %.3f\n", result.Key, result.Score)
}

Combine vector search with metadata filtering: 

filter := &qdb.FilterCondition{
    Field:    "category",
    Operator: "eq",
    Value:    "document",
}

results, err := db.SearchVector(ctx, query, 5, filter)

Configuration

Customize QDB behavior with configuration options: 

config := &qdb.Config{
    DatabasePath: "my-database.db",
    UseMemory:    false, // Use file storage
    HNSWConfig: &qdb.HNSWConfig{
        M:              16,
        EfConstruction: 200,
        EfSearch:       50,
        MaxLayers:      6,
        Ml:             1.0 / math.Log(2.0),
    },
}

db, err := qdb.New(config)

Package qdb provides a high-performance in-memory database with vector search capabilities.

QDB combines Redis-style key-value operations with advanced vector search, all implemented in pure Go without external dependencies.

Features

- Redis-like O(1) hash table operations - Vector search using HNSW (Hierarchical Navigable Small World) algorithm - Exact K-nearest neighbors search - Hybrid queries with filtering - Support for multiple data types (string, int, float64, bool, []byte, slices, maps) - Thread-safe concurrent operations - File-based or in-memory storage

Quick Start 

package main

import (
    "context"
    "log"
    "github.com/mew-sh/qdb"
)

func main() {
    ctx := context.Background()

    // Create database instance
    config := qdb.DefaultConfig()
    db, err := qdb.New(config)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    // Basic key-value operations
    db.Set(ctx, "key", "value")
    result := db.Get(ctx, "key")
    value, _ := result.String()

    // Vector operations
    vector := []float64{1.0, 2.0, 3.0, 4.0}
    metadata := map[string]interface{}{"category": "test"}
    db.SetVector(ctx, "vec1", vector, metadata)

    query := []float64{1.1, 2.1, 3.1, 4.1}
    results, _ := db.SearchVector(ctx, query, 5)
}

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrKeyNotFound       = errors.New("key not found")
	ErrInvalidVector     = errors.New("invalid vector")
	ErrDimensionMismatch = errors.New("vector dimension mismatch")
	ErrEmptyIndex        = errors.New("index is empty")
	ErrInvalidFilter     = errors.New("invalid filter condition")
	ErrStorageClosed     = errors.New("storage is closed")
)

Common errors

Functions

This section is empty.

Types

type Config 

type Config struct {
	DatabasePath string
	HNSWConfig   *HNSWConfig
	UseMemory    bool // If true, uses in-memory storage instead of file
}

Config holds configuration options for QDB instances. It allows customization of storage backend, HNSW parameters, and operational modes.

func DefaultConfig 

func DefaultConfig() *Config

DefaultConfig returns a default configuration DefaultConfig returns a Config with sensible default values for most use cases. The default configuration uses file-based storage with balanced HNSW parameters optimized for both accuracy and performance.

type FileStorage 

type FileStorage struct {
	// contains filtered or unexported fields
}

FileStorage handles file-based persistence

func NewFileStorage 

func NewFileStorage(filePath string, useMemory bool) (*FileStorage, error)

NewFileStorage creates a new file storage instance

func (*FileStorage) Close 

func (fs *FileStorage) Close() error

Close closes the file storage

func (*FileStorage) Delete 

func (fs *FileStorage) Delete(key string) error

Delete removes a record from storage

func (*FileStorage) Load 

func (fs *FileStorage) Load(key string) (*StorageRecord, error)

Load retrieves a record by key from storage

func (*FileStorage) LoadAll 

func (fs *FileStorage) LoadAll() ([]*StorageRecord, error)

LoadAll loads all records from storage

func (*FileStorage) Store 

func (fs *FileStorage) Store(record *StorageRecord) error

Store saves a record to storage

type FilterCondition 

type FilterCondition struct {
	Field    string      `json:"field"`
	Operator string      `json:"operator"` // "eq", "ne", "gt", "lt", "gte", "lte", "in"
	Value    interface{} `json:"value"`
}

FilterCondition represents a filter condition for search FilterCondition defines a filtering condition for vector search operations. It allows filtering vectors based on their metadata using various operators such as equality, comparison, and membership tests.

type HNSW 

type HNSW struct {
	// contains filtered or unexported fields
}

HNSW implements Hierarchical Navigable Small World graphs

func NewHNSW 

func NewHNSW(config *HNSWConfig) *HNSW

NewHNSW creates a new HNSW instance

func (*HNSW) Insert 

func (h *HNSW) Insert(id int, vector []float64) error

Insert adds a new vector to the HNSW index

func (*HNSW) Search 

func (h *HNSW) Search(query []float64, k int) ([]int, []float64, error)

Search performs ANN search in the HNSW index

type HNSWConfig 

type HNSWConfig struct {
	M              int     // Number of bi-directional links created for every new element during construction
	EfConstruction int     // Size of the dynamic candidate list
	EfSearch       int     // Size of the dynamic candidate list used during search
	MaxLayers      int     // Maximum number of layers
	Ml             float64 // Level generation parameter
}

HNSWConfig holds HNSW algorithm parameters

type HNSWNode 

type HNSWNode struct {
	ID          int
	Vector      []float64
	Level       int
	Connections map[int][]int // layer -> list of connected node IDs
}

HNSWNode represents a node in the HNSW graph

type HashBucket 

type HashBucket struct {
	// contains filtered or unexported fields
}

HashBucket represents a bucket in the hash table

type HashEntry 

type HashEntry struct {
	Key   string
	Value interface{}
	Next  *HashEntry
}

HashEntry represents an entry in a hash bucket

type HashTable 

type HashTable struct {
	// contains filtered or unexported fields
}

HashTable implements a Redis-style hash table with O(1) operations

func NewHashTable 

func NewHashTable() *HashTable

NewHashTable creates a new hash table

func (*HashTable) Delete 

func (ht *HashTable) Delete(key string)

Delete removes a key-value pair from the hash table

func (*HashTable) Get 

func (ht *HashTable) Get(key string) (interface{}, bool)

Get retrieves a value from the hash table

func (*HashTable) Set 

func (ht *HashTable) Set(key string, value interface{})

Set stores a key-value pair in the hash table

type QDB 

type QDB struct {
	// contains filtered or unexported fields
}

QDB represents the main database instance that provides both key-value storage and vector search capabilities. It combines a Redis-style hash table for O(1) key-value operations with HNSW-based vector indexing for efficient similarity search.

QDB is thread-safe and supports concurrent operations through fine-grained locking. It can operate in memory-only mode for maximum performance or with file-based persistence for data durability.

func New 

func New(config *Config) (*QDB, error)

New creates a new QDB instance with the specified configuration. It initializes the hash table, vector database, and storage backend. If config is nil, DefaultConfig() is used.

Returns an error if the storage backend cannot be initialized or if existing data cannot be loaded.

func (*QDB) Close 

func (q *QDB) Close() error

Close closes the database

func (*QDB) Delete 

func (q *QDB) Delete(ctx context.Context, key string) error

Delete removes a key-value pair

func (*QDB) Exists 

func (q *QDB) Exists(ctx context.Context, key string) (bool, error)

Exists checks if a key exists

func (*QDB) Get 

func (q *QDB) Get(ctx context.Context, key string) *Result

Get retrieves a value by key

func (*QDB) SearchVector 

func (q *QDB) SearchVector(ctx context.Context, query []float64, k int, filters ...*FilterCondition) ([]*SearchResult, error)

SearchVector performs ANN search using HNSW

func (*QDB) SearchVectorExact 

func (q *QDB) SearchVectorExact(ctx context.Context, query []float64, k int, filters ...*FilterCondition) ([]*SearchResult, error)

SearchVectorExact performs exact K-NN search

func (*QDB) Set 

func (q *QDB) Set(ctx context.Context, key string, value interface{}) error

Set stores a key-value pair

func (*QDB) SetVector 

func (q *QDB) SetVector(ctx context.Context, key string, vector []float64, metadata interface{}) error

SetVector stores a vector with optional metadata

type Result 

type Result struct {
	// contains filtered or unexported fields
}

Result represents a query result Result wraps the result of a Get operation, providing type-safe methods to extract values in various formats. It implements a fluent interface for easy value conversion and error handling.

func (*Result) Bytes 

func (r *Result) Bytes() ([]byte, error)

Bytes returns the value as bytes

func (*Result) Float 

func (r *Result) Float() (float64, error)

Float returns the value as a float64

func (*Result) Int 

func (r *Result) Int() (int, error)

Int returns the value as an integer

func (*Result) Result 

func (r *Result) Result() (interface{}, error)

Result returns the value and error

func (*Result) String 

func (r *Result) String() (string, error)

String returns the value as a string

type SearchResult 

type SearchResult struct {
	Key      string      `json:"key"`
	Vector   []float64   `json:"vector"`
	Score    float64     `json:"score"`
	Metadata interface{} `json:"metadata"`
}

SearchResult represents a search result SearchResult represents a single result from a vector search operation. It contains the key, matching vector, similarity score, and associated metadata.

type StorageRecord 

type StorageRecord struct {
	Key       string      `json:"key"`
	Value     interface{} `json:"value"`
	DataType  string      `json:"data_type"`
	CreatedAt time.Time   `json:"created_at"`
	UpdatedAt time.Time   `json:"updated_at"`
}

StorageRecord represents a record in storage

type Vector 

type Vector struct {
	ID        int         `json:"id"`
	Key       string      `json:"key"`
	Data      []float64   `json:"data"`
	Dimension int         `json:"dimension"`
	Metadata  interface{} `json:"metadata"`
	CreatedAt time.Time   `json:"created_at"`
}

Vector represents a vector with metadata

type VectorDB 

type VectorDB struct {
	// contains filtered or unexported fields
}

VectorDB handles vector storage and search operations

func NewVectorDB 

func NewVectorDB(storage *FileStorage, config *HNSWConfig) *VectorDB

NewVectorDB creates a new vector database

func (*VectorDB) SearchANN 

func (vdb *VectorDB) SearchANN(query []float64, k int, filters ...*FilterCondition) ([]*SearchResult, error)

SearchANN performs approximate nearest neighbor search using HNSW

func (*VectorDB) SearchExact 

func (vdb *VectorDB) SearchExact(query []float64, k int, filters ...*FilterCondition) ([]*SearchResult, error)

SearchExact performs exact K-NN search

func (*VectorDB) SetVector 

func (vdb *VectorDB) SetVector(key string, vector []float64, metadata interface{}) error

SetVector stores a vector in the vector database

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.