quelldb

README

QuellDB

QuellDB is a lightweight, embeddable, high-performance key-value store written in Go.
Built on a Log-Structured Merge Tree (LSM Tree) architecture, QuellDB provides:

• Fast in-memory writes
• Durable disk-based persistence
• Optional AES-256 encryption
• Binary-safe, compressed SSStorages using Snappy
• Range-aware compaction with manifest tracking
• Simple, pluggable Go API

---

Features

• In-memory MemStorage with thread-safe access
• Persistent SSStorages (Sorted String Storages)
• Optional AES-256 encryption with GCM mode
• Snappy compression by default (even without encryption)
• Write-Ahead Log (WAL) for durability before flush
• Bloom filter support for efficient lookups
• TTL (Time-To-Live) support for expiring keys
• Batch writes via PutBatch()
• Key iteration via Iterator(), with prefix filters
• Versioned manifest system
• Range-aware SSStorage compaction based on overlapping key ranges

---

Installation

go get github.com/quellington/quelldb

---

Quick Start

Snappy compression (Default)

package main

import (
    "fmt"
    "github.com/quellington/quelldb"
)

func main() {
    store, err := quelldb.Open("data", nil)
    if err != nil {
        panic(err)
    }

    store.Put("username", "john")
    store.Flush()

    val, _ := store.Get("username")
    fmt.Println("Value:", val)
}

With AES-256 Encryption (If needed)

package main

import (
    "fmt"
    "github.com/quellington/quelldb"
)

func main() {
    key := []byte("thisis32byteslongthisis32byteslo") // 32 bytes

    store, err := quelldb.Open("securedata", &quelldb.Options{
        EncryptionKey: key,
    })
    if err != nil {
        panic(err)
    }

    store.Put("user:123", `{"id":"123","username":"john","email":"john@example.io","age":50}`)
    store.Flush()

    val, _ := store.Get("user:123")
    fmt.Println("Decrypted:", val)
}

TTL Support

store.PutTTL("temp:session", "expires-soon", 10*time.Second)

Batch Writes

store.PutBatch(map[string]string{
    "user:1": "alice",
    "user:2": "bob",
})

Prefix Iteration

it := store.PrefixIterator("user:")
for it.Next() {
    fmt.Println(it.Key(), it.Value())
}

Encryption Logic

If you provide a 32-byte EncryptionKey, QuellDB will:

• Compress values with Snappy
• Encrypt them using AES-256 GCM mode
• Without a key, values are still compressed and unreadable to humans, but not encrypted

API Reference

Function	Description
Open()	Initializes a database at given path
Put(key, val)	Writes data into memory and WAL
Get(key)	Retrieves value from memory or SSStorages
Delete(key)	Deletes a key from memory and appends DEL to WAL
Flush()	Persists current MemStorage to a new SSStorage
PutBatch(map[string]string)	PeWrites multiple key-value pairs in one WAL flush
PutTTL(key, val, ttl)	Writes a key with an expiration duration
Iterator()	Iterates all sorted keys from memory
PrefixIterator(p)	Iterates sorted keys with the given prefix
Compact(p)	Compacts overlapping SSStorage into a single one
Subscribe(func(ChangeEvent)) int	Registers a live event handler and returns a handler ID

MIT License © 2025 The QuellDB Authors

Documentation

Index

• func EncodeManifest(ssts []SSSMeta, key []byte) ([]byte, error)
• func SaveManifest(basePath string, ssts []SSSMeta, key []byte) error
• type ChangeEvent
• type DB
  • func Open(path string, opts *Options) (*DB, error)
  • func (db *DB) Close() error
  • func (db *DB) Compact() error
  • func (db *DB) Delete(key string) error
  • func (db *DB) Flush() error
  • func (db *DB) Get(key string) (string, error)
  • func (db *DB) Iterator() *Iterator
  • func (db *DB) PrefixIterator(prefix string) *Iterator
  • func (db *DB) Put(key, value string) error
  • func (db *DB) PutBatch(kvs map[string]string) error
  • func (db *DB) PutTTL(key, value string, ttl time.Duration) error
  • func (db *DB) Subscribe(handler func(ChangeEvent)) int
  • func (db *DB) Unsubscribe(id int)
• type Iterator
  • func (it *Iterator) Key() string
  • func (it *Iterator) Next() bool
  • func (it *Iterator) Value() string
• type Options
• type SSSMeta
  • func DecodeManifest(data []byte, key []byte) ([]SSSMeta, error)
  • func LoadManifest(basePath string, key []byte) ([]SSSMeta, error)

Functions

func EncodeManifest

func EncodeManifest(ssts []SSSMeta, key []byte) ([]byte, error)

EncodeManifest encodes SSStorage names with binary format, nappy, ptional encryption

func SaveManifest

func SaveManifest(basePath string, ssts []SSSMeta, key []byte) error

SaveManifest writes a new numbered manifest and updates CURRENT

Types

type ChangeEvent

type ChangeEvent struct {
	Type  string
	Key   string
	Value string
}

type DB

type DB struct {
	// contains filtered or unexported fields
}

func Open

func Open(path string, opts *Options) (*DB, error)

Open initializes a new database at the specified path. It creates the necessary directories and initializes the WAL. If the path already exists, it will be opened for reading. The options parameter allows for optional encryption key setup. If the encryption key is provided, it must be 32 bytes long for AES-256. If the key is not provided, the database will be unencrypted. The function returns a pointer to the DB instance and an error if any occurs.

func (*DB) Close

func (db *DB) Close() error

Close closes the database and the WAL. It ensures that all data is flushed to the SSS files and the WAL is closed properly. The function returns an error if any occurs during the close operation. It is important to call this function when you are done using the database to ensure data integrity. The function does not delete any SSS files or the WAL file. It only closes the file handles and ensures that all data is written to disk. After calling this function, the database instance should not be used anymore.

func (*DB) Compact

func (db *DB) Compact() error

Compact merges multiple SSStorage into a single one. It reads all SSStorages in the base path, merges them into a single map, and writes the merged data into a new SSStorage file. The old SSStorage files are deleted after the merge.

func (*DB) Delete

func (db *DB) Delete(key string) error

Delete removes the key-value pair associated with the given key. It first deletes the pair from memory and then writes the delete operation to the WAL. The function returns an error if any occurs during the write operation. If the key does not exist, it will not raise an error. The function does not check the SSS files for the key before deleting it from memory. If the key is found in memory, it will be deleted immediately.

func (*DB) Flush

func (db *DB) Flush() error

Flush writes the in-memory data to a new SSS file. It generates a new filename based on the highest existing SSS file number. The new file will be named with the format "sss_00001.qldb". The function returns an error if any occurs during the write operation.

func (*DB) Get

func (db *DB) Get(key string) (string, error)

Get retrieves the value associated with the given key. It first checks the in-memory storage and then searches through the SSS files in reverse order. The function returns the value and error whether the key was found. If the key is not found in memory or in any SSS files, it returns an empty string and false. If the key is found, the value is returned in plaintext. If encryption is enabled, the value will be decrypted before returning. The function also handles the case where the key is not found in any SSS files. If the key is found in memory, it will be returned immediately without checking the SSS files.

func (*DB) Iterator

func (db *DB) Iterator() *Iterator

NewIterator creates a new iterator for the database. It retrieves all keys from the in-memory storage, sorts them, and initializes the iterator with the sorted keys and their corresponding values. The iterator starts at index -1, indicating that it is before the first element. The caller can use the Next() method to advance the iterator and access keys and values. The iterator is not thread-safe and should be used by a single goroutine at a time. The caller is responsible for closing the iterator when done. The iterator does not require any additional resources to be closed.

func (*DB) PrefixIterator

func (db *DB) PrefixIterator(prefix string) *Iterator

NewPrefixIterator creates a new iterator for the database with a specific prefix. It retrieves all keys from the in-memory storage that start with the given prefix, sorts them, and initializes the iterator with the sorted keys and their corresponding values.

func (*DB) Put

func (db *DB) Put(key, value string) error

Put stores a key-value pair in the database. It first stores the pair in memory and then writes it to the WAL. The function returns an error if any occurs during the write operation. If the key already exists, it will be updated with the new value. The value is stored in plaintext, and if encryption is enabled, it will be encrypted before writing to the WAL.

func (*DB) PutBatch

func (db *DB) PutBatch(kvs map[string]string) error

PutBatch stores multiple key-value pairs in the database. It first stores the pairs in memory and then writes them to the WAL. If the key already exists, it will be updated with the new value.

func (*DB) PutTTL

func (db *DB) PutTTL(key, value string, ttl time.Duration) error

PutWithTTL stores a key-value pair in the database with a specified time-to-live (TTL). It first stores the pair in memory with the specified TTL and then writes it to the WAL. The function takes a key, a value, and a TTL duration as parameters. The TTL duration specifies how long the key-value pair should be valid. After the TTL expires, the key-value pair will be automatically removed from the in-memory storage. The function returns an error if any occurs during the write operation.

func (*DB) Subscribe

func (db *DB) Subscribe(handler func(ChangeEvent)) int

Subscribe allows you to register a callback function that will be called whenever a change event occurs in the database. The callback function receives a ChangeEvent struct containing the type of change, the key, and the value.

func (*DB) Unsubscribe

func (db *DB) Unsubscribe(id int)

Unsubscribe removes a subscriber from the database. The subscriber is identified by its ID, which is returned when the subscriber was added.

type Iterator

type Iterator struct {
	// contains filtered or unexported fields
}

func (*Iterator) Key

func (it *Iterator) Key() string

func (*Iterator) Next

func (it *Iterator) Next() bool

Next advances the iterator to the next key-value pair.

func (*Iterator) Value

func (it *Iterator) Value() string

type Options

type Options struct {
	EncryptionKey []byte
	CompactLimit  uint
	BoomBitSize   uint
	BoomHashCount uint
}

type SSSMeta

type SSSMeta struct {
	Filename string
	MinKey   string
	MaxKey   string
}

func DecodeManifest

func DecodeManifest(data []byte, key []byte) ([]SSSMeta, error)

DecodeManifest decodes manifest data to extract SSStorage names

func LoadManifest

func LoadManifest(basePath string, key []byte) ([]SSSMeta, error)

LoadManifest reads CURRENT, then loads the correct numbered manifest