README
¶
appendstore
appendstore is a small, embeddable append-only key/value store for Go. It
provides crash recovery, checksummed records, optional in-memory secondary
indexes, an LRU value cache, and atomic compaction using only the Go standard
library.
The module is intended for applications that need a durable local store owned by one process. It is not a distributed database or a transactional SQL engine.
Install
go get github.com/sirrobot01/appendstore
Usage
package main
import (
"fmt"
"log"
"github.com/sirrobot01/appendstore"
)
func main() {
store, err := appendstore.Open("records.db", appendstore.Options{
IndexedFields: []string{"kind"},
})
if err != nil {
log.Fatal(err)
}
defer store.Close()
err = store.Put("greeting", []byte("hello"), &appendstore.PutOptions{
Attributes: map[string]string{"kind": "message"},
})
if err != nil {
log.Fatal(err)
}
value, err := store.Get("greeting")
if err != nil {
log.Fatal(err)
}
fmt.Println(string(value))
fmt.Println(store.KeysBy("kind", "message"))
}
Process model
Only one process may open a database path at a time. Open acquires an
exclusive operating-system lock and returns ErrStoreLocked when another
process owns the store. The adjacent .lock file remains after Close by
design; the operating-system lock, not the file's presence, indicates
ownership. Locks are released automatically when a process exits.
Calls on one Store are safe for concurrent use by multiple goroutines.
Durability
Each mutation is appended immediately. Options.SyncInterval controls when
data is synchronized to durable storage:
0synchronizes every mutation.- A positive duration synchronizes periodically.
- A negative duration disables automatic synchronization, including on
Close; callSyncexplicitly when needed.
On startup, appendstore rebuilds its indexes, discards an incomplete trailing record, and rejects checksum corruption. Compaction writes and synchronizes a replacement before atomically installing it. Compaction blocks all reads and writes until the replacement is installed; the pause grows with the size of the live data.
Log format
New records are always written in the version 5 format. Version 5 is designed
to be the last format change that breaks compatibility. docs/stable-format.md
specifies it in full.
Two mechanisms replace the version number for later changes:
- Feature bits. The header holds an incompatible mask and a read-only mask.
OpenreturnsErrUnsupportedFeaturefor an unknown incompatible bit, and names the bit. For an unknown read-only bit,Opensucceeds,ReadOnly()reportstrue, and every write returns an error that wrapsErrReadOnly. - Record extensions. Each record ends with an extension area of tagged entries. A reader steps over a tag it does not know, and compaction copies the entry unchanged. A later release can therefore add a per-record field that an older reader ignores, without a feature bit and without a new version. The reader and compaction sides are in place. There is no public API to write an extension yet, so the area is empty in every record this release writes.
Open still reads legacy version 1–4 logs and migrates them:
- A version 4 log needs only a new header, because its records are already valid version 5 records.
- A version 1–3 log is rewritten in one atomic compaction.
Open returns ErrUnsupportedVersion for a log newer than version 5.
Format migration
Migration is one-way. A build compiled against the older format rejects a
migrated log with ErrUnsupportedVersion, so an application that upgrades and
then rolls back cannot read its own data.
To keep a way back, Open copies the log to <path>.v<version>.bak before it
migrates. A backup that cannot be written fails the open and leaves the log
untouched. An existing backup is never replaced, so a repeated attempt cannot
overwrite the original with partly migrated data. Backups are never reclaimed;
delete them when the older release is no longer a target.
Set Options.OnMigrate to observe the migration. It runs after the backup is
taken and before the log is rewritten, and an error from it aborts the open:
store, err := appendstore.Open(path, appendstore.Options{
OnMigrate: func(info appendstore.MigrationInfo) error {
log.Printf("upgrading %s from v%d to v%d; restore %s to roll back",
info.Path, info.FromVersion, info.ToVersion, info.Backup)
return nil
},
})
Set Options.NoMigrationBackup to skip the copy. Use it only where the
application has its own way back, or where the disk cannot hold a second copy
of the log.
To roll back, stop the application and restore each backup over its store:
mv store.db.v3.bak store.db
A backup only returns the data that existed when the store was migrated. To roll back a store that has been written since, use the downgrade tool. It writes a version 3 or version 4 log from the live contents of a version 5 log:
go run ./cmd/appendstore-downgrade -to 4 store.db store-v4.db
The tool refuses to write anything when a record holds data the target format cannot express. A version 4 target cannot express a record extension. A version 3 target also cannot express an attribute outside its fixed set of named fields.
The tool refuses a destination that already exists. Pass -f to replace one;
the output is renamed into place only after every record is written, so a failed
run leaves the old file untouched.
The tool reads the source log. It writes to the source only to discard an
incomplete trailing record from an interrupted write, which is the same repair
that Open performs.
A version 4 log holds the same attribute map as version 5, so that target is lossless. A version 3 log holds a fixed field per attribute instead, so a value that was absent comes back as an empty field when the log is read again.
The downgrade tool exists for the transition to version 5. It is removed together with the version 1–4 migration path.
Limits
- All keys and index metadata stay in memory. The store is not suitable for keyspaces that exceed available RAM.
- Each value is encoded in one in-memory buffer on write and read. Very large values increase memory use accordingly (hard limit: 2 GiB per value).
- Secondary indexes support exact-match equality only; there are no range or prefix queries.
Status
The on-disk format is versioned, but the public API should be considered pre-1.0. Back up important data before upgrading between early releases.
Documentation
¶
Overview ¶
Package appendstore provides a high-performance append-only log storage engine with in-memory indexing, LRU caching, and secondary indexes.
Architecture:
- Append-only log file for durability
- In-memory primary index for O(1) key lookups
- LRU cache for frequently accessed entries
- Memory-mapped reads with a file-read fallback
- Optional secondary indexes over application-defined attributes
- Background compaction to reclaim deleted space
Thread Safety:
- All operations are thread-safe via RWMutex
- Reads can proceed concurrently
- Writes are serialized
- Open acquires an exclusive process lock for the database path
Durability:
- All writes are appended to the log immediately
- Index is rebuilt from log on startup (crash recovery)
- Optional periodic sync for fsync guarantees
Example ¶
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/sirrobot01/appendstore"
)
func main() {
dir, err := os.MkdirTemp("", "appendstore-example-")
if err != nil {
panic(err)
}
defer os.RemoveAll(dir)
store, err := appendstore.Open(filepath.Join(dir, "example.db"), appendstore.Options{
IndexedFields: []string{"kind"},
})
if err != nil {
panic(err)
}
defer store.Close()
if err := store.Put("greeting", []byte("hello"), &appendstore.PutOptions{
Attributes: map[string]string{"kind": "message"},
}); err != nil {
panic(err)
}
value, err := store.Get("greeting")
if err != nil {
panic(err)
}
fmt.Println(string(value))
fmt.Println(store.KeysBy("kind", "message"))
}
Output: hello [greeting]
Index ¶
- Variables
- func Downgrade(src, dst string, options DowngradeOptions) error
- type DowngradeOptions
- type Metadata
- type MigrationInfo
- type Options
- type PutOptions
- type Stats
- type Store
- func (s *Store) AttributeValues(field string) []string
- func (s *Store) Close() error
- func (s *Store) Compact() error
- func (s *Store) CountBy(field, value string) int
- func (s *Store) Delete(key string) error
- func (s *Store) DiskSize() int64
- func (s *Store) Exists(key string) bool
- func (s *Store) ForEach(fn func(key string, value []byte) error) error
- func (s *Store) ForEachMetadata(fn func(key string, meta *Metadata) error) error
- func (s *Store) Get(key string) ([]byte, error)
- func (s *Store) GetMetadata(key string) (*Metadata, error)
- func (s *Store) Keys() []string
- func (s *Store) KeysBy(field, value string) []string
- func (s *Store) Len() int
- func (s *Store) MemoryUsage() int64
- func (s *Store) NeedsCompaction() bool
- func (s *Store) Put(key string, value []byte, options *PutOptions) error
- func (s *Store) ReadOnly() bool
- func (s *Store) Stats() Stats
- func (s *Store) Sync() error
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrStoreClosed = errors.New("store is closed") ErrStoreLocked = errors.New("store is already open by another process") ErrCorruptedData = errors.New("corrupted data detected") ErrUnsupportedVersion = errors.New("unsupported log version") ErrUnsupportedFeature = errors.New("unsupported log feature") ErrReadOnly = errors.New("store is read-only") ErrCompactionInProgress = errors.New("compaction already in progress") ErrKeyNotFound = errors.New("key not found") ErrInvalidKey = errors.New("invalid key") ErrInvalidAttribute = errors.New("invalid attribute name") ErrValueTooLarge = errors.New("value too large") ErrAttributesTooLarge = errors.New("attributes too large") )
Common errors
Functions ¶
func Downgrade ¶ added in v0.6.0
func Downgrade(src, dst string, options DowngradeOptions) error
Downgrade writes the live contents of the log at src to a new log at dst in format version 3 or version 4. It is the way back for a caller that upgraded to version 5 and must run an older build again.
Downgrade fails, and leaves nothing at dst, when a record holds data the target format cannot express: an extension entry, an attribute version 3 has no field for, or a value too long for a version 3 field.
Downgrade reads src. It writes to src only to discard an incomplete trailing record from an interrupted write, which is the same repair Open performs.
This function exists for the transition described in docs/stable-format.md. It is removed together with the migration path when support for versions 1 to 4 ends.
Types ¶
type DowngradeOptions ¶ added in v0.6.0
type DowngradeOptions struct {
// Version is the target log format: 3 or 4.
Version uint32
// Overwrite replaces a destination that already exists. Without it,
// Downgrade refuses rather than replace a file it did not create. The
// replacement is still atomic: the output is renamed into place only after
// every record is written, so a failed run leaves the old file untouched.
Overwrite bool
}
DowngradeOptions configures Downgrade.
type Metadata ¶
Metadata is the application-defined metadata associated with a value. Its Attributes map is always a defensive copy owned by the caller.
type MigrationInfo ¶ added in v0.5.0
type MigrationInfo struct {
// Path is the store path being migrated.
Path string
// FromVersion is the format found on disk; ToVersion is what replaces it.
FromVersion, ToVersion uint32
// Backup is the copy of the original log, or "" when NoMigrationBackup is
// set. Report it: it is what a downgrade restores.
Backup string
}
MigrationInfo describes a log about to be rewritten in the current format.
type Options ¶
type Options struct {
// CacheSize is the maximum number of entries to cache (default: 1000)
CacheSize int
// SyncInterval controls automatic fsync behavior: 0 syncs every mutation,
// a positive duration syncs periodically, and a negative duration disables
// automatic syncing (including on Close). Sync can always be called explicitly.
SyncInterval time.Duration
// CompactionThreshold is the deleted entry ratio that triggers compaction (default: 0.2)
CompactionThreshold float64
// AutoCompact enables automatic background compaction
AutoCompact bool
// IndexedFields lists attribute names that should have in-memory secondary
// indexes. Only these fields can be queried with KeysBy efficiently.
IndexedFields []string
// OnError receives errors from background sync and compaction. It is never
// called while the store mutex is held.
OnError func(error)
// OnMigrate is called when Open finds a log in an older format, after the
// backup is taken and before the log is rewritten. Returning an error
// aborts Open and leaves the log in its original format. Optional.
OnMigrate func(MigrationInfo) error
// NoMigrationBackup skips the copy Open keeps of a pre-migration log.
// Migrating is one-way — a build that predates the new format refuses to
// read a migrated log — so only set this where the caller has its own way
// back, or where the disk cannot spare a second copy of the log.
NoMigrationBackup bool
}
Options configures a Store.
type PutOptions ¶
PutOptions describes application-defined metadata for a value.
type Stats ¶
type Stats struct {
Writes int64
Reads int64
CacheHits int64
CacheMisses int64
Deletes int64
Compactions int64
}
Stats is an instantaneous snapshot of operational counters. Reads counts every value returned by Get and ForEach, whether served from the cache or from disk; CacheHits and CacheMisses partition the Get calls.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an append-only storage engine.
func (*Store) AttributeValues ¶
AttributeValues returns the sorted values present for an indexed field.
func (*Store) ForEach ¶
ForEach iterates over all entries in disk order (optimized for sequential I/O). The value passed to fn is owned by ForEach and only valid for the duration of that call — callers that retain it must copy.
Scans deliberately bypass the LRU cache: a full pass touches every entry once and never reuses them, so caching would evict the genuinely-hot working set (active streams) and replace it with single-use scan data. Each value is read under a short RLock so writers interleave between keys and reads observe the current log even across a compaction swap. A single reusable buffer keeps the scan allocation-free per record.
func (*Store) ForEachMetadata ¶
ForEachMetadata iterates over a snapshot of metadata only (no disk reads).
func (*Store) GetMetadata ¶
GetMetadata retrieves metadata without reading the full value. This is O(1) from the in-memory index - no disk access
func (*Store) MemoryUsage ¶
MemoryUsage returns approximate in-memory usage
func (*Store) NeedsCompaction ¶
NeedsCompaction returns true if compaction would reclaim significant space
func (*Store) Put ¶
func (s *Store) Put(key string, value []byte, options *PutOptions) error
Put stores a key-value pair with optional application-defined attributes.
func (*Store) ReadOnly ¶ added in v0.6.0
ReadOnly reports whether Open put the store in read-only mode because the log sets a read-only feature bit this build does not know. Every write then returns an error wrapping ErrReadOnly that names the feature.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
appendstore-downgrade
command
Command appendstore-downgrade writes an older log from an appendstore database.
|
Command appendstore-downgrade writes an older log from an appendstore database. |