README
¶
appendstore
appendstore is a small, embeddable append-only key/value store for Go. It
provides crash recovery, checksummed records, optional in-memory secondary
indexes, an LRU value cache, and atomic compaction using only the Go standard
library.
The module is intended for applications that need a durable local store owned by one process. It is not a distributed database or a transactional SQL engine.
Install
go get github.com/sirrobot01/appendstore
Usage
package main
import (
"fmt"
"log"
"github.com/sirrobot01/appendstore"
)
func main() {
store, err := appendstore.Open("records.db", appendstore.Options{
IndexedFields: []string{"kind"},
})
if err != nil {
log.Fatal(err)
}
defer store.Close()
err = store.Put("greeting", []byte("hello"), &appendstore.PutOptions{
Attributes: map[string]string{"kind": "message"},
})
if err != nil {
log.Fatal(err)
}
value, err := store.Get("greeting")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(value))
fmt.Println(store.KeysBy("kind", "message"))
}
Process model
Only one process may open a database path at a time. Open acquires an
exclusive operating-system lock and returns ErrStoreLocked when another
process owns the store. The adjacent .lock file remains after Close by
design; the operating-system lock, not the file's presence, indicates
ownership. Locks are released automatically when a process exits.
Calls on one Store are safe for concurrent use by multiple goroutines.
Durability
Each mutation is appended immediately. Options.SyncInterval controls when
data is synchronized to durable storage:
0synchronizes every mutation.- A positive duration synchronizes periodically.
- A negative duration disables automatic synchronization, including on
Close; callSyncexplicitly when needed.
On startup, appendstore rebuilds its indexes, discards an incomplete trailing record, and rejects checksum corruption. Compaction writes and synchronizes a replacement before atomically installing it. Compaction blocks all reads and writes until the replacement is installed; the pause grows with the size of the live data.
New records are always written in the version 4 format. Open reads legacy
version 1–3 logs and migrates them to version 4 in one atomic compaction
before it returns. Open returns ErrUnsupportedVersion for logs newer than
version 4.
Format migration
Migration is one-way. A build compiled against the older format rejects a
migrated log with ErrUnsupportedVersion, so an application that upgrades and
then rolls back cannot read its own data.
To keep a way back, Open copies the log to <path>.v<version>.bak before it
migrates. A backup that cannot be written fails the open and leaves the log
untouched. An existing backup is never replaced, so a repeated attempt cannot
overwrite the original with partly migrated data. Backups are never reclaimed;
delete them when the older release is no longer a target.
Set Options.OnMigrate to observe the migration. It runs after the backup is
taken and before the log is rewritten, and an error from it aborts the open:
store, err := appendstore.Open(path, appendstore.Options{
OnMigrate: func(info appendstore.MigrationInfo) error {
log.Printf("upgrading %s from v%d to v%d; restore %s to roll back",
info.Path, info.FromVersion, info.ToVersion, info.Backup)
return nil
},
})
Set Options.NoMigrationBackup to skip the copy. Use it only where the
application has its own way back, or where the disk cannot hold a second copy
of the log.
To roll back, stop the application and restore each backup over its store:
mv store.db.v3.bak store.db
Limits
- All keys and index metadata stay in memory. The store is not suitable for keyspaces that exceed available RAM.
- Each value is encoded in one in-memory buffer on write and read. Very large values increase memory use accordingly (hard limit: 2 GiB per value).
- Secondary indexes support exact-match equality only; there are no range or prefix queries.
Status
The on-disk format is versioned, but the public API should be considered pre-1.0. Back up important data before upgrading between early releases.
Documentation
¶
Overview ¶
Package appendstore provides a high-performance append-only log storage engine with in-memory indexing, LRU caching, and secondary indexes.
Architecture:
- Append-only log file for durability
- In-memory primary index for O(1) key lookups
- LRU cache for frequently accessed entries
- Memory-mapped reads with a file-read fallback
- Optional secondary indexes over application-defined attributes
- Background compaction to reclaim deleted space
Thread Safety:
- All operations are thread-safe via RWMutex
- Reads can proceed concurrently
- Writes are serialized
- Open acquires an exclusive process lock for the database path
Durability:
- All writes are appended to the log immediately
- Index is rebuilt from log on startup (crash recovery)
- Optional periodic sync for fsync guarantees
Example ¶
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/sirrobot01/appendstore"
)
func main() {
dir, err := os.MkdirTemp("", "appendstore-example-")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)
store, err := appendstore.Open(filepath.Join(dir, "example.db"), appendstore.Options{
IndexedFields: []string{"kind"},
})
if err != nil {
panic(err)
}
defer store.Close()
if err := store.Put("greeting", []byte("hello"), &appendstore.PutOptions{
Attributes: map[string]string{"kind": "message"},
}); err != nil {
panic(err)
}
value, err := store.Get("greeting")
if err != nil {
panic(err)
}
fmt.Println(string(value))
fmt.Println(store.KeysBy("kind", "message"))
}
Output: hello [greeting]
Index ¶
- Variables
- type Metadata
- type MigrationInfo
- type Options
- type PutOptions
- type Stats
- type Store
- func (s *Store) AttributeValues(field string) []string
- func (s *Store) Close() error
- func (s *Store) Compact() error
- func (s *Store) CountBy(field, value string) int
- func (s *Store) Delete(key string) error
- func (s *Store) DiskSize() int64
- func (s *Store) Exists(key string) bool
- func (s *Store) ForEach(fn func(key string, value []byte) error) error
- func (s *Store) ForEachMetadata(fn func(key string, meta *Metadata) error) error
- func (s *Store) Get(key string) ([]byte, error)
- func (s *Store) GetMetadata(key string) (*Metadata, error)
- func (s *Store) Keys() []string
- func (s *Store) KeysBy(field, value string) []string
- func (s *Store) Len() int
- func (s *Store) MemoryUsage() int64
- func (s *Store) NeedsCompaction() bool
- func (s *Store) Put(key string, value []byte, options *PutOptions) error
- func (s *Store) Stats() Stats
- func (s *Store) Sync() error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrStoreClosed = errors.New("store is closed") ErrStoreLocked = errors.New("store is already open by another process") ErrCorruptedData = errors.New("corrupted data detected") ErrUnsupportedVersion = errors.New("unsupported log version") ErrCompactionInProgress = errors.New("compaction already in progress") ErrKeyNotFound = errors.New("key not found") ErrInvalidKey = errors.New("invalid key") ErrInvalidAttribute = errors.New("invalid attribute name") ErrValueTooLarge = errors.New("value too large") ErrAttributesTooLarge = errors.New("attributes too large") )
Common errors
Functions ¶
This section is empty.
Types ¶
type Metadata ¶
Metadata is the application-defined metadata associated with a value. Its Attributes map is always a defensive copy owned by the caller.
type MigrationInfo ¶ added in v0.5.0
type MigrationInfo struct {
// Path is the store path being migrated.
Path string
// FromVersion is the format found on disk; ToVersion is what replaces it.
FromVersion, ToVersion uint32
// Backup is the copy of the original log, or "" when NoMigrationBackup is
// set. Report it: it is what a downgrade restores.
Backup string
}
MigrationInfo describes a log about to be rewritten in the current format.
type Options ¶
type Options struct {
// CacheSize is the maximum number of entries to cache (default: 1000)
CacheSize int
// SyncInterval controls automatic fsync behavior: 0 syncs every mutation,
// a positive duration syncs periodically, and a negative duration disables
// automatic syncing (including on Close). Sync can always be called explicitly.
SyncInterval time.Duration
// CompactionThreshold is the deleted entry ratio that triggers compaction (default: 0.2)
CompactionThreshold float64
// AutoCompact enables automatic background compaction
AutoCompact bool
// IndexedFields lists attribute names that should have in-memory secondary
// indexes. Only these fields can be queried with KeysBy efficiently.
IndexedFields []string
// OnError receives errors from background sync and compaction. It is never
// called while the store mutex is held.
OnError func(error)
// OnMigrate is called when Open finds a log in an older format, after the
// backup is taken and before the log is rewritten. Returning an error
// aborts Open and leaves the log in its original format. Optional.
OnMigrate func(MigrationInfo) error
// NoMigrationBackup skips the copy Open keeps of a pre-migration log.
// Migrating is one-way — a build that predates the new format refuses to
// read a migrated log — so only set this where the caller has its own way
// back, or where the disk cannot spare a second copy of the log.
NoMigrationBackup bool
}
Options configures a Store.
type PutOptions ¶
PutOptions describes application-defined metadata for a value.
type Stats ¶
type Stats struct {
Writes int64
Reads int64
CacheHits int64
CacheMisses int64
Deletes int64
Compactions int64
}
Stats is an instantaneous snapshot of operational counters. Reads counts every value returned by Get and ForEach, whether served from the cache or from disk; CacheHits and CacheMisses partition the Get calls.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an append-only storage engine.
func (*Store) AttributeValues ¶
AttributeValues returns the sorted values present for an indexed field.
func (*Store) ForEach ¶
ForEach iterates over all entries in disk order (optimized for sequential I/O). The value passed to fn is owned by ForEach and only valid for the duration of that call — callers that retain it must copy.
Scans deliberately bypass the LRU cache: a full pass touches every entry once and never reuses them, so caching would evict the genuinely-hot working set (active streams) and replace it with single-use scan data. Each value is read under a short RLock so writers interleave between keys and reads observe the current log even across a compaction swap. A single reusable buffer keeps the scan allocation-free per record.
func (*Store) ForEachMetadata ¶
ForEachMetadata iterates over a snapshot of metadata only (no disk reads).
func (*Store) GetMetadata ¶
GetMetadata retrieves metadata without reading the full value. This is O(1) from the in-memory index - no disk access
func (*Store) MemoryUsage ¶
MemoryUsage returns approximate in-memory usage
func (*Store) NeedsCompaction ¶
NeedsCompaction returns true if compaction would reclaim significant space
func (*Store) Put ¶
func (s *Store) Put(key string, value []byte, options *PutOptions) error
Put stores a key-value pair with optional application-defined attributes.
Source Files