wildcat

package module
v2.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2026 License: MPL-2.0 Imports: 24 Imported by: 0

README

License GitHub Release

Wildcat is a high-performance embedded key-value database (or storage engine) written in Go with C interoperability. It incorporates modern database design principles including LSM (Log-Structured Merge) tree architecture, MVCC (Multi-Version Concurrency Control), and lock-free data structures for its critical paths, along with automatic background operations to deliver excellent read/write performance with immediate consistency and durability.

Features

  • LSM (Log-Structured Merge) tree architecture optimized for write-heavy workloads
  • Mostly lock-free MVCC with minimal blocking on critical paths
  • WAL logging captures full transaction state for recovery and rehydration
  • Version-aware skip list for fast in-memory MVCC access
  • Thread-safe write operations with atomic coordination
  • Scalable design with background flusher and compactor
  • Concurrent block storage leveraging direct, offset-based file I/O (using pread/pwrite) for optimal performance
  • Atomic LRU cache for active block manager handles
  • Atomic memtable lifecycle management
  • SSTables are immutable BTrees
  • Configurable durability levels None (fastest), Partial (balanced), Full (most durable)
  • Snapshot-isolated MVCC with timestamp-based reads
  • Crash recovery preserves committed transactions and maintains access to incomplete transactions if configured with RecoverUncommittedTxns
  • Automatic multi-threaded background compaction with configurable concurrency
  • ACID transaction support with configurable durability guarantees
  • Range, prefix, and full iteration support with bidirectional traversal
  • High transactional throughput per second with low latency due to lock-free and non-blocking design from memory to disk
  • Optional Bloom filters per SSTable for improved key lookup performance
  • Key-value separation optimization (.klog for keys, .vlog for values)
  • Tombstone-aware and version-aware compaction with retention based on active transaction read windows
  • Transaction recovery preserves incomplete transactions for post-crash inspection and resolution if db configured with RecoverUncommittedTxns
  • Keys and values stored as opaque byte sequences
  • Single-node embedded storage engine with no network or replication overhead

Discord Community

Join our Discord community to discuss development, design, ask questions, and get help with Wildcat.

Discord

Table of Contents

Samples

You can find sample programs here

Benchmarks

You can find the benchmarking tool here

Wiki

You can read about Wildcat's architecture at the wiki.

Version and Compatibility

  • Go 1.24+
  • Linux/macOS/Windows (64-bit)

Basic Usage

Wildcat supports opening multiple *wildcat.DB instances in parallel, each operating independently in separate directories.

Downloading

go get github.com/wildcatdb/wildcat/v2

Specific major version i.e v2.x.x can be downloaded using

go get github.com/wildcatdb/wildcat/v2@v2.x.x

Import

import (
    "github.com/wildcatdb/wildcat/v2"
)

When importing different majors you can do

// v1 (not recommended, use v2+)
import (
    "github.com/wildcatdb/wildcat"
)

// v2
import (
    "github.com/wildcatdb/wildcat/v2"
)

// v2+
import (
    "github.com/wildcatdb/wildcat/v32"
)

Opening a Wildcat DB instance

The only required option is the database directory path.

// Create default options
opts := &wildcat.Options{
    Directory: "/path/to/db",
    // You don't need to set all options only Directory is required!
}

// Open or create a new Wildcat DB instance
db, err := wildcat.Open(opts) // Returns *wildcat.DB
if err != nil {
    // Handle error
}
defer db.Close()

Simple Key-Value Operations

The easiest way to interact with Wildcat is through the Update method, which handles transactions automatically. This means it runs begin, commit, and rollback for you, allowing you to focus on the operations themselves.

// Write a value
err := db.Update(func(txn *wildcat.Txn) error {
    return txn.Put([]byte("hello"), []byte("world")) // Put update's existing key's values
})
if err != nil {
    // Handle error
}

// Read a value
var result []byte
err = db.View(func(txn *wildcat.Txn) error {
    var err error
    result, err = txn.Get([]byte("hello"))
    return err
})

if err != nil {
    // Handle error
} else {
    fmt.Println("Value:", string(result)) // Outputs: Value: world
}

Manual Transaction Management

For more complex operations, you can manually manage transactions.

txn, err := db.Begin()
if err != nil {
    // Handle error
    return
}

// Perform operations
err := txn.Put([]byte("key1"), []byte("value1"))
if err != nil {
    txn.Rollback()
    // Handle error
}

value, err := txn.Get([]byte("key1"))
if err != nil {
    txn.Rollback()
    // Handle error
}

// Commit or rollback
err = txn.Commit()
if err != nil {
    txn.Rollback()
    // Handle commit error
}

Read-Only Transactions with View

// Read a value with View
var result []byte
err = db.View(func(txn *wildcat.Txn) error {
    var err error
    result, err = txn.Get([]byte("hello"))
    return err
})

Batch Operations

You can perform multiple operations in a single transaction.

[!CAUTION] Batch operations on the Wildcat engine are slower completed inside an Update. It's better to use Begin->Put flow for batch writes.

err := db.Update(func(txn *wildcat.Txn) error {
    // Write multiple key-value pairs
    for i := 0; i < 1000; i++ {
        key := []byte(fmt.Sprintf("key%d", i))
        value := []byte(fmt.Sprintf("value%d", i))

        if err := txn.Put(key, value); err != nil {
            return err
        }
    }
    return nil
})

OR

// Perform batch operations
for i := 0; i < 1000; i++ {
    txn, err := db.Begin()
    if err != nil {
        // Handle error
        return
    }

    key := []byte(fmt.Sprintf("key%d", i))
    value := []byte(fmt.Sprintf("value%d", i))

    if err := txn.Put(key, value); err != nil {
        txn.Rollback()
        // Handle error
        return
    }

    err = txn.Commit()
    if err != nil {
        // Handle error
        return
    }
}

Iterating Keys

Wildcat provides comprehensive iteration capabilities with MVCC consistency.

[!TIP] You can set ascending or descending order, and iterate over all keys, a range of keys, or keys with a specific prefix.

Full Iterator (bidirectional)
err := db.View(func(txn *wildcat.Txn) error {
    // Create ascending iterator
    iter, err := txn.NewIterator(true)
    if err != nil {
        return err
    }

    // Iterate forward
    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    // Change direction and iterate backward
    err = iter.SetDirection(false)
    if err != nil {
        return err
    }

    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    return nil
})
Range Iterator (bidirectional)
err := db.View(func(txn *wildcat.Txn) error {
    iter, err := txn.NewRangeIterator([]byte("start"), []byte("end"), true)
    if err != nil {
        return err
    }

    // Iterate forward
    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    // Change direction and iterate backward
    err = iter.SetDirection(false)
    if err != nil {
        return err
    }

    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    return nil
})
Prefix Iterator (bidirectional)
err := db.View(func(txn *wildcat.Txn) error {
    iter, err := txn.NewPrefixIterator([]byte("prefix"), true)
    if err != nil {
        return err
    }

    // Iterate forward
    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    // Change direction and iterate backward
    err = iter.SetDirection(false)
    if err != nil {
        return err
    }

    for {
        key, value, timestamp, ok := iter.Next()
        if !ok {
            break
        }

        fmt.Printf("Key: %s, Value: %s, Timestamp: %d\n", key, value, timestamp)
    }

    return nil
})

Transaction Recovery

// After reopening a database, you can access recovered transactions
txn, err := db.GetTxn(transactionID)
if err != nil {
    // Transaction not found or error
    return err
}

// Inspect the recovered transaction state
fmt.Printf("Transaction %d status: committed=%v\n", txn.Id, txn.Committed)
fmt.Printf("Write set: %v\n", txn.WriteSet)
fmt.Printf("Delete set: %v\n", txn.DeleteSet)

// You can commit or rollback the recovered transaction
if !txn.Committed {
    err = txn.Commit() // or txn.Rollback()
}

Log Channel

Wildcat provides a log channel for real-time logging. You can set up a goroutine to listen for log messages.

// Create a log channel
logChannel := make(chan string, 100) // Buffer size of 100 messages

// Set up options with the log channel
opts := &wildcat.Options{
    Directory:       "/path/to/db",
    LogChannel:      logChannel,
    // Other options...
}

// Open the database
db, err := wildcat.Open(opts)
if err != nil {
    // Handle error
}

wg := &sync.WaitGroup{}

wg.Add(1)

// Start a goroutine to listen to the log channel
go func() {
    defer wg.Done()
    for msg := range logChannel {
        // Process log messages
        fmt.Println("wildcat:", msg)

        // You could also write to a file, send to a logging service, etc.
        // log.Println(msg)
    }
}()

// Use..

wg.Wait() // Wait for the goroutine to finish

// When you're done, close the database
defer db.Close()

Database Statistics

Wildcat provides comprehensive statistics about database state

stats := db.Stats()
fmt.Println(stats)

Output example

┌───────────────────────────────────────────────────────────────────────────┐
│ Wildcat DB Stats and Configuration                                        │
├───────────────────────────────────────────────────────────────────────────┤
│ Write Buffer Size          : 25                                           │
│ Sync Option                : 1                                            │
│ Level Count                : 6                                            │
│ Bloom Filter Enabled       : false                                        │
│ Max Compaction Concurrency : 4                                            │
│ Compaction Cooldown        : 5s                                           │
│ Compaction Batch Size      : 8                                            │
│ Compaction Size Ratio      : 1.1                                          │
│ Compaction Threshold       : 8                                            │
│ Score Size Weight          : 0.8                                          │
│ Score Count Weight         : 0.2                                          │
│ Flusher Interval           : 1ms                                          │
│ Compactor Interval         : 250ms                                        │
│ Bloom FPR                  : 0.01                                         │
│ WAL Retry                  : 10                                           │
│ WAL Backoff                : 128µs                                        │
│ SSTable B-Tree Order       : 10                                           │
│ LRU Size                   : 1024                                         │
│ File Version               : 2                                            │
│ Magic Number               : 1464421444                                   │
│ Directory                  : /tmp/wildcat_stats_example/                  │
├───────────────────────────────────────────────────────────────────────────┤
│ ID Generator State                                                        │
├───────────────────────────────────────────────────────────────────────────┤
│ Last SST ID                : 5                                            │
│ Last WAL ID                : 11                                           │
├───────────────────────────────────────────────────────────────────────────┤
│ Runtime Statistics                                                        │
├───────────────────────────────────────────────────────────────────────────┤
│ Active Memtable Size       : 0                                            │
│ Active Memtable Entries    : 0                                            │
│ Active Transactions        : 0                                            │
│ WAL Files                  : 5                                            │
│ Total SSTables             : 4                                            │
│ Total Entries              : 20                                           │
└───────────────────────────────────────────────────────────────────────────┘

This returns detailed information including

  • Configuration settings and tuning parameters
  • Active memtable size and entry count
  • Transaction counts and oldest active read timestamp
  • Level statistics and SSTable counts
  • ID generator states and WAL file counts
  • Compaction and flushing statistics

Force Flushing

You can force a flush of current and immutable memtables in queue to disk using the Flush method.

// Force all memtables to flush to SSTables
err := db.ForceFlush()
if err != nil {
    // Handle error
}

Escalate Sync

If you have your sync option set to SyncNone and would like to control when the block manager syncs a WAL to disk, you can use the *DB.Sync() which syncs the current WAL to disk.

// Escalate sync to ensure current WAL is written to disk
err := db.Sync()
if err != nil {
    // Handle error
}

Advanced Configuration

Wildcat provides many configuration options for fine-tuning.

Parameter Example Value Description
Directory "/path/to/database" The path where the database files will be stored
WriteBufferSize 64 * 1024 * 1024 Size threshold for memtable before flushing to disk
SyncOption wildcat.SyncFull Controls durability vs performance tradeoff
SyncNone - Fastest, but no durability guarantees
SyncPartial - Balances performance and durability
SyncFull - Maximum durability, slower performance
SyncInterval 128 * time.Millisecond Time between background sync operations (only for SyncPartial)
LevelCount 7 Number of levels in the LSM tree
LevelMultiplier 10 Size ratio between adjacent levels
BlockManagerLRUSize 1024 Number of block managers to cache
SSTableBTreeOrder 10 Size of SSTable klog block sets
LogChannel make(chan string, 1000) Channel for real-time logging, useful for debugging and monitoring
BloomFilter false Enable or disable bloom filters for SSTables to speed up key lookups. Bloom filters use double hashing with FNV-1a and FNV hash functions. Is automatically sized based on expected items and desired false positive rate.
MaxCompactionConcurrency 4 Maximum number of concurrent compactions
CompactionCooldownPeriod 5 * time.Second Cooldown period between compactions to prevent thrashing
CompactionBatchSize 8 Max number of SSTables to compact at once
CompactionSizeRatio 1.1 Level size ratio that triggers compaction
CompactionSizeThreshold 8 Number of files to trigger size-tiered compaction
CompactionScoreSizeWeight 0.8 Weight for size-based compaction scoring
CompactionScoreCountWeight 0.2 Weight for count-based compaction scoring
CompactionSizeTieredSimilarityRatio 1.5 Similarity ratio for size-tiered compaction. For grouping SSTables that are "roughly the same size" together for compaction.
CompactionActiveSSTReadWaitBackoff 8 * time.Microsecond Backoff is used to avoid busy waiting when checking if sstables are safe to remove during compaction process final steps
CompactionPartitionRatio 0.6 How much to move back (0.6 = 60% of data)
CompactionPartitionDistributionRatio 0.7 How to split between L-1 and L-2 (0.7 = 70% to L-1, 30% to L-2)
FlusherTickerInterval 1 * time.Millisecond Interval for flusher background process
CompactorTickerInterval 250 * time.Millisecond Interval for compactor background process
BloomFilterFPR 0.01 False positive rate for Bloom filters
WalAppendRetry 10 Number of retries for WAL append operations
WalAppendBackoff 128 * time.Microsecond Backoff duration for WAL append retries
STDOutLogging false If true, logs will be printed to stdout instead of the log channel. Log channel will be ignored if provided.
MaxConcurrentTxns 65536 Maximum number of concurrent transactions. This is the size of the ring buffer used for transaction management.
TxnBeginRetry 10 Number of retries for Begin() when the transaction buffer is full.
TxnBeginBackoff 1 * time.Microsecond Initial backoff duration for Begin() retries when the transaction buffer is full.
TxnBeginMaxBackoff 100 * time.Millisecond Maximum backoff duration for Begin() retries when the transaction buffer is full.
RecoverUncommittedTxns true If true, Wildcat will attempt to recover uncommitted transactions on startup. This allows you to inspect and potentially commit or rollback transactions that were in progress at the time of a crash.

Shared C Library

You will require the latest Go toolchain to build the shared C library for Wildcat. This allows you to use Wildcat as a C library in other languages.

Building the C Library

# Linux x64
GOOS=linux GOARCH=amd64 go build -buildmode=c-shared -o libwildcat.so wildcat_c.go

# Linux ARM64
GOOS=linux GOARCH=arm64 go build -buildmode=c-shared -o libwildcat.so wildcat_c.go

# Windows x64
GOOS=windows GOARCH=amd64 go build -buildmode=c-shared -o wildcat.dll wildcat_c.go

# Windows ARM64
GOOS=windows GOARCH=arm64 go build -buildmode=c-shared -o wildcat.dll wildcat_c.go

# macOS x64 (Intel)
GOOS=darwin GOARCH=amd64 go build -buildmode=c-shared -o libwildcat.dylib wildcat_c.go

# macOS ARM64 (Apple Silicon)
GOOS=darwin GOARCH=arm64 go build -buildmode=c-shared -o libwildcat.dylib wildcat_c.go

# FreeBSD x64
GOOS=freebsd GOARCH=amd64 go build -buildmode=c-shared -o libwildcat.so wildcat_c.go

# FreeBSD ARM64
GOOS=freebsd GOARCH=arm64 go build -buildmode=c-shared -o libwildcat.so wildcat_c.go

C API

For C example check c/example.c in the repository.

Linux instructions

Once you've built your shared library, you can use the below commands.

sudo cp libwildcat.so /usr/local/lib/
sudo cp libwildcat.h /usr/local/include/
sudo ldconfig

Now you can include header

#include <wildcat.h>

Then you can link against the library when compiling. Below is an example compiling example.c in repository on Linux.

gcc -o wildcat_example c/example.c -L. -lwildcat -lpthread
Options
typedef struct {
    char* directory;
    long write_buffer_size;
    int sync_option;
    long sync_interval_ns;
    int level_count;
    int level_multiplier;
    int block_manager_lru_size;
    int permission;
    int bloom_filter;
    int max_compaction_concurrency;
    long compaction_cooldown_ns;
    int compaction_batch_size;
    double compaction_size_ratio;
    int compaction_size_threshold;
    double compaction_score_size_weight;
    double compaction_score_count_weight;
    double compaction_size_tiered_similarity_ratio;
    long flusher_interval_ns;
    long compactor_interval_ns;
    double bloom_fpr;
    int wal_append_retry;
    long wal_append_backoff_ns;
    int sstable_btree_order;
    int stdout_logging;
    int max_compaction_concurrency;
    int txn_begin_retry;
    long txn_begin_backoff_ns;
    long txn_begin_max_backoff_ns;
    int recover_uncommitted_txns;
    double partition_ratio;
    double partition_distribution_ratio;
} wildcat_opts_t;
Sync Options
typedef enum {
    SYNC_NONE = 0,
    SYNC_FULL,
    SYNC_PARTIAL
} sync_option_t;
C API Functions
extern long unsigned int wildcat_open(wildcat_opts_t* opts);
extern void wildcat_close(long unsigned int handle);
extern long int wildcat_begin_txn(long unsigned int handle);
extern int wildcat_txn_put(long unsigned int handle, long int txnId, char* key, char* val);
extern char* wildcat_txn_get(long unsigned int handle, long int txnId, char* key);
extern int wildcat_txn_delete(long unsigned int handle, long int txnId, char* key);
extern int wildcat_txn_commit(long unsigned int handle, long int txnId);
extern int wildcat_txn_rollback(long unsigned int handle, long int txnId);
extern void wildcat_txn_free(long unsigned int handle, long int txnId);
extern long unsigned int wildcat_txn_new_iterator(long unsigned int handle, long int txnId, int asc);
extern long unsigned int wildcat_txn_new_range_iterator(long unsigned int handle, long int txnId, char* start, char* end, int asc);
extern long unsigned int wildcat_txn_new_prefix_iterator(long unsigned int handle, long int txnId, char* prefix, int asc);
extern char* wildcat_stats(long unsigned int handle);
extern int wildcat_force_flush(long unsigned int handle);
extern int wildcat_txn_iterate_next(long unsigned int id);
extern int wildcat_txn_iterate_prev(long unsigned int id);
extern int wildcat_txn_iter_valid(long unsigned int id);
extern char* wildcat_iterator_key(long unsigned int id);
extern char* wildcat_iterator_value(long unsigned int id);
extern void wildcat_iterator_free(long unsigned int id);
extern int wildcat_sync(long unsigned int handle);

Contributing

You are open to contribute to Wildcat. Find a bug, optimization, refactor and submit a PR. It will be reviewed. You're only helping us all.

Documentation

Overview

Package wildcat

(C) Copyright Alex Gaetano Padula

Licensed under the Mozilla Public License, v. 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.mozilla.org/en-US/MPL/2.0/

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const (
	SSTablePrefix     = "sst_"     // Prefix for SSTable files
	LevelPrefix       = "l"        // Prefix for level directories i.e. "l1", "l2", etc.
	WALFileExtension  = ".wal"     // Extension for Write Ahead Log files <id>.wal
	KLogExtension     = ".klog"    // Extension for KLog files
	VLogExtension     = ".vlog"    // Extension for VLog files
	IDGSTFileName     = "idgstate" // Filename for ID generator state
	TempFileExtension = ".tmp"     // Temporary file extension for intermediate files
)

Prefixes, filenames, extensions constants

View Source 
const (
	FarFutureOffsetNs              = 10_000_000_000 // 10 seconds in nanoseconds; added to time.Now().UnixNano() to scan all versions
	TombstoneBlockID         int64 = -1             // Sentinel ValueBlockID indicating a deletion marker in KLogEntry
	FlushTargetLevel               = 1              // Memtables are always flushed to level 1 (L0 is the active memtable)
	CompactionScoreThreshold       = 1.0            // Compaction score above which a level triggers compaction
	PartitioningJobPriority        = 10.0           // Highest priority value for last-level partitioning jobs
	MinSSTablesToCompact           = 2              // Minimum number of SSTables required to perform a compaction
	SizeTieredMaxLevelIdx          = 2              // Levels with index < this use size-tiered compaction; others use leveled
	MinLevelsForPartitioning       = 2              // Minimum last-level index required before partitioning is considered
	MinBTreeOrder                  = 2              // Minimum allowed B-tree order for SSTables
	BytesPerMB                     = 1024 * 1024    // Bytes in a megabyte, used for human-readable log output
	JitterFraction                 = 4              // Divides backoff to produce ±25% jitter
	JitterCoinFlip                 = 2              // Used with rand.Intn for 50/50 jitter direction
)

Internal constants used across the codebase to avoid magic numbers

View Source 
const (
	DefaultWriteBufferSize = 64 * 1024 * 1024     // Default write buffer size
	DefaultSyncOption      = SyncNone             // Default sync option for write operations
	DefaultSyncInterval    = 16 * time.Nanosecond // Default sync interval for write operations
	DefaultLevelCount      = 6                    // Default number of levels in the LSM tree
	DefaultLevelMultiplier = 10                   // Multiplier for the number of levels
	// 64MB -> 640MB -> 6.4GB ->  64GB -> 640GB ->  6.4TB
	DefaultBlockManagerLRUSize                  = 1024                 // Size of the LRU cache for block managers
	DefaultPermission                           = 0750                 // Default permission for created files
	DefaultMaxCompactionConcurrency             = 4                    // Default max compaction concurrency
	DefaultCompactionCooldownPeriod             = 5 * time.Second      // Default cooldown period for compaction
	DefaultCompactionBatchSize                  = 8                    // Default max number of SSTables to compact at once
	DefaultCompactionSizeRatio                  = 1.1                  // Default level size ratio that triggers compaction
	DefaultCompactionSizeThreshold              = 8                    // Default number of files to trigger size-tiered compaction
	DefaultCompactionScoreSizeWeight            = 0.8                  // Default weight for size-based score
	DefaultCompactionScoreCountWeight           = 0.2                  // Default weight for count-based score
	DefaultCompactionSizeTieredSimilarityRatio  = 1.5                  // Default similarity ratio for size-tiered compaction
	DefaultCompactionActiveSSTReadWaitBackoff   = 8 * time.Microsecond // Backoff is used to avoid busy waiting when checking if sstables are safe to remove during compaction process final steps
	DefaultFlusherTickerInterval                = 1 * time.Millisecond
	DefaultCompactorTickerInterval              = 250 * time.Millisecond // Default interval for compactor ticker
	DefaultBloomFilterFPR                       = 0.01                   // Default false positive rate for Bloom filter
	DefaultWALAppendRetry                       = 10                     // Default number of retries for WAL append
	DefaultWALAppendBackoff                     = 128 * time.Microsecond // Default backoff duration for WAL append
	DefaultSSTableBTreeOrder                    = 10                     // Default order of the B-tree for SSTables
	DefaultMaxConcurrentTxns                    = 65536                  // Default max concurrent transactions
	DefaultTxnBeginRetry                        = 10                     // Default retries for Begin()
	DefaultTxnBeginBackoff                      = 1 * time.Microsecond   // Default initial backoff
	DefaultTxnBeginMaxBackoff                   = 100 * time.Millisecond // Default max backoff
	DefaultCompactionPartitionRatio             = 0.2                    // Move 20% of data back when partitioning
	DefaultCompactionPartitionDistributionRatio = 0.7                    // 70% to L-1, 30% to L-2
)

Defaults

Variables

This section is empty.

Functions

This section is empty.

Types

type Compactor 

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor is responsible for managing compaction jobs

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents the main Wildcat structure

func Open 

func Open(opts *Options) (*DB, error)

Open initializes a new Wildcat instance with the provided options

func (*DB) Begin 

func (db *DB) Begin() (*Txn, error)

Begin starts a new transaction

func (*DB) Close 

func (db *DB) Close() error

Close closes the database and all open resources

func (*DB) ForceFlush 

func (db *DB) ForceFlush() error

ForceFlush forces the flush of main current memtable and immutable memtables in flusher queue

func (*DB) GetTxn 

func (db *DB) GetTxn(id int64) (*Txn, error)

GetTxn retrieves a transaction by ID. Can be used on system recovery. You can recover an incomplete transaction.

func (*DB) Stats 

func (db *DB) Stats() string

Stats returns a string with the current statistics of the Wildcat database

func (*DB) Sync 

func (db *DB) Sync() error

Sync escalates the Fdatasync operation for the current memtable's write ahead log. To only be used when sync option is set to SyncNone.

func (*DB) Update 

func (db *DB) Update(fn func(txn *Txn) error) error

Update performs an atomic update using a transaction

func (*DB) View 

func (db *DB) View(fn func(txn *Txn) error) error

View performs a read-only transaction

type Flusher 

type Flusher struct {
	// contains filtered or unexported fields
}

Flusher is responsible for queuing and flushing memtables to disk

type IDGType 

type IDGType int64
const (
	// IDGTypeInt64 represents an int64 ID generator
	IDGTypeInt64 IDGType = iota
	// IDGTypeTimestamp represents a timestamp-based ID generator
	IDGTypeTimestamp
)

type IDGenerator 

type IDGenerator struct {
	// contains filtered or unexported fields
}

IDGenerator is a thread-safe ID generator

type IDGeneratorState 

type IDGeneratorState struct {
	LastSstID int64 // Last SSTable ID
	LastWalID int64 // Last WAL ID
	// contains filtered or unexported fields
}

IDGeneratorState represents the state of the ID generator. When system shuts down the state is saved to disk and restored on next startup.

type KLogEntry 

type KLogEntry struct {
	Key          []byte // Key of the entry
	Timestamp    int64  // Timestamp of the entry
	ValueBlockID int64  // Block ID of the value
}

KLogEntry represents a key-value entry in the KLog

type Level 

type Level struct {
	// contains filtered or unexported fields
}

Level is a disk level within Wildcat, which contains a list of immutable SSTables

func (*Level) SSTables 

func (l *Level) SSTables() []*SSTable

SSTables returns the list of SSTables in the level

type Memtable 

type Memtable struct {
	// contains filtered or unexported fields
}

Memtable is a memory table structure

type MergeIterator 

type MergeIterator struct {
	// contains filtered or unexported fields
}

MergeIterator combines multiple iterators into a single iterator

func NewMergeIterator 

func NewMergeIterator(db *DB, iterators []*iterator, ts int64, ascending bool) (*MergeIterator, error)

NewMergeIterator creates a new MergeIterator with the given iterators

func (*MergeIterator) Close added in v2.3.9 

func (mi *MergeIterator) Close()

Close cleans up resources and returns iterators to the pool

func (*MergeIterator) HasNext 

func (mi *MergeIterator) HasNext() bool

HasNext returns true if there are more entries in the configured direction

func (*MergeIterator) HasPrev 

func (mi *MergeIterator) HasPrev() bool

HasPrev returns true if there are entries in the opposite direction

func (*MergeIterator) IsAscending 

func (mi *MergeIterator) IsAscending() bool

IsAscending returns the current iteration direction

func (*MergeIterator) Next 

func (mi *MergeIterator) Next() ([]byte, []byte, int64, bool)

Next returns the next key-value pair in the configured direction Returns slices that reference existing data to avoid allocations

func (*MergeIterator) Prev 

func (mi *MergeIterator) Prev() ([]byte, []byte, int64, bool)

Prev returns the previous key-value pair (opposite of configured direction)

func (*MergeIterator) SetDirection 

func (mi *MergeIterator) SetDirection(ascending bool) error

SetDirection changes the iteration direction

type Options 

type Options struct {
	Directory                            string        // Directory for Wildcat
	WriteBufferSize                      int64         // Size of the write buffer
	SyncOption                           SyncOption    // Sync option for write operations
	SyncInterval                         time.Duration // Interval for syncing the write buffer
	LevelCount                           int           // Number of levels in the LSM tree
	LevelMultiplier                      int           // Multiplier for the number of levels
	BlockManagerLRUSize                  int           // Size of the LRU cache for block managers
	Permission                           os.FileMode   // Permission for created files
	LogChannel                           chan string   // Channel for logging
	STDOutLogging                        bool          // Enable logging to standard output (default is false and if set, channel is ignored)
	BloomFilter                          bool          // Enable Bloom filter for SSTables
	MaxCompactionConcurrency             int           // Maximum number of concurrent compactions
	CompactionCooldownPeriod             time.Duration // Cooldown period for compaction
	CompactionBatchSize                  int           // Max number of SSTables to compact at once
	CompactionSizeRatio                  float64       // Level size ratio that triggers compaction
	CompactionSizeThreshold              int           // Number of files to trigger size-tiered compaction
	CompactionScoreSizeWeight            float64       // Weight for size-based score
	CompactionScoreCountWeight           float64       // Weight for count-based score
	CompactionSizeTieredSimilarityRatio  float64       // Similarity ratio for size-tiered compaction.  For grouping SSTables that are "roughly the same size" together for compaction.
	CompactionActiveSSTReadWaitBackoff   time.Duration // Backoff time for active SSTable read wait during compaction, to avoid busy waiting
	CompactionPartitionRatio             float64       // How much to move back (0.6 = 60% of data). Used for last level compaction
	CompactionPartitionDistributionRatio float64       // How to split between L-1 and L-2 (0.7 = 70% to L-1, 30% to L-2)
	CompactorTickerInterval              time.Duration // Interval for compactor ticker
	FlusherTickerInterval                time.Duration // Interval for flusher ticker
	BloomFilterFPR                       float64       // False positive rate for Bloom filter
	WalAppendRetry                       int           // Number of retries for WAL append
	WalAppendBackoff                     time.Duration // Backoff duration for WAL append
	SSTableBTreeOrder                    int           // Order of the B-tree for SSTables
	MaxConcurrentTxns                    int           // Maximum concurrent transactions (buffer size)
	TxnBeginRetry                        int           // Number of retries for Begin() when buffer full
	TxnBeginBackoff                      time.Duration // Initial backoff duration for Begin() retries
	TxnBeginMaxBackoff                   time.Duration // Maximum backoff duration for Begin() retries
	RecoverUncommittedTxns               bool          // Whether to recover uncommitted transactions on startup
}

Options represents the configuration options for Wildcat

type SSTable 

type SSTable struct {
	Id          int64                    // SStable ID
	Min         []byte                   // The minimum key in the SSTable
	Max         []byte                   // The maximum key in the SSTable
	Size        int64                    // The size of the SSTable in bytes
	EntryCount  int                      // The number of entries in the SSTable
	Level       int                      // The level of the SSTable
	BloomFilter *bloomfilter.BloomFilter // Optional bloom filter for fast lookups
	Timestamp   int64                    // Timestamp of latest entry in the SSTable
	// contains filtered or unexported fields
}

SSTable represents a sorted string table

type SyncOption 

type SyncOption int

SyncOption is a block manager sync option that can be set for a *DB instance. the sync option will be used for all block managers created by the *DB instance internally. 

const (
	SyncNone    SyncOption = iota // We don't sync
	SyncFull                      // We sync the entire block manager after each write
	SyncPartial                   // We sync based at intervals based on the SyncInterval option.. So every SyncInterval we sync the block manager.
)

type Txn 

type Txn struct {
	Id        int64             // The transactions id, can be recovered
	ReadSet   map[string]int64  // Key -> Timestamp
	WriteSet  map[string][]byte // Key -> Value
	DeleteSet map[string]bool   // Key -> Deleted
	Timestamp int64             // The timestamp of the transaction
	Committed bool              // Whether the transaction is committed
	// contains filtered or unexported fields
}

Txn represents a transaction in a Wildcat DB instance

func (*Txn) Commit 

func (txn *Txn) Commit() error

Commit commits the transaction

func (*Txn) Delete 

func (txn *Txn) Delete(key []byte) error

Delete removes a key from database

func (*Txn) Get 

func (txn *Txn) Get(key []byte) ([]byte, error)

Get retrieves a value by key

func (*Txn) NewIterator 

func (txn *Txn) NewIterator(asc bool) (*MergeIterator, error)

NewIterator creates a new bidirectional iterator

func (*Txn) NewPrefixIterator 

func (txn *Txn) NewPrefixIterator(prefix []byte, asc bool) (*MergeIterator, error)

NewPrefixIterator creates a new prefix bidirectional iterator

func (*Txn) NewRangeIterator 

func (txn *Txn) NewRangeIterator(startKey []byte, endKey []byte, asc bool) (*MergeIterator, error)

NewRangeIterator creates a new range bidirectional iterator

func (*Txn) Put 

func (txn *Txn) Put(key []byte, value []byte) error

Put adds key-value pair to database

func (*Txn) Rollback 

func (txn *Txn) Rollback() error

Rollback rolls back the transaction

type WAL 

type WAL struct {
	// contains filtered or unexported fields
}

WAL is a write-ahead log structure

Source Files

View all Source files

Directories

Path Synopsis
Package blockmanager
 Package blockmanager
Package bloomfilter
 Package bloomfilter
Package buffer
 Package buffer
Package lru
 Package lru
Package queue
 Package queue
Package skiplist
 Package skiplist
Package tree
 Package tree

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.