beachdb

module

v0.0.5 Latest Latest Go to latest Published: May 29, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/aalhour/beachdb

Links

Open Source Insights

README ¶

BeachDB is a toy distributed NoSQL database. Built for learning and education, not production.

It starts life as a small, inspectable storage engine, then deliberately grows “real-system bones”: a server API, a failure model, and a Raft-replicated core. The point isn’t to win benchmarks — it’s to understand, measure, and explain what’s actually happening.

Backstory

I’ve been fond of distributed systems and databases for a long time. I wrote my first Hadoop and Apache Spark pipeline back in 2016, then went on to solve hairy stream-processing problems at Shopify, and later worked on Apache HBase at HubSpot where I helped build and operate database infrastructure on top of Kubernetes at massive scale.

BeachDB is my attempt to re-learn the fundamentals by building them from scratch in Go. I’m prioritizing simplicity, clarity, and understanding over scalability, speed, and micro-optimizations.

Architecture

LSM storage engine (WAL → memtable → SSTables → compaction)
Single-node API (server wrapper for Get/Put/Delete/Scan with timeouts + backpressure)
Distributed replication with Raft (single group: leader writes + leader reads; log entry == WriteBatch)
Inspectability-first (dump tools + crash tests as part of the architecture)

Key features (shipped as a checklist)

This list is ordered to match the build + blog sequence. I’ll tick these off as they land.

Engine (storage truth)

Scope + semantics contract (snapshots, iterators, durability), see: intro blog post
WAL v1: checksums + deterministic crash recovery (fsync per committed batch), see: durability blog post
Memtable v1: sorted structure + tombstones, see: memtable blog post
Reference-model randomized tests (model vs implementation)
SSTables v1: immutable sorted files + sst_dump, see: sstables blog post
Crash-loop harness: kill mid-write, reopen, validate invariants, see: crash-testing, part 1 blog post
Manifest v1: durable SSTable catalog (CURRENT + VersionEdit log) for startup reconstruction + manifest_dump, see: manifest blog post
Merge iterators (memtable + SSTs) + snapshot reads (seqno-based)
Read path acceleration: block index + bloom filters + benchmark evidence
Compaction v1: one strategy, minimal knobs + amplification measurements
Adversarial testing: fault injection + fuzzing (WAL/SST decode paths)

Server (systems truth)

Binary protocol (framed) + timeouts + backpressure
Load generator + p50/p99 latency reporting
Metrics/tracing hooks that make performance explainable

Replication (distributed truth)

Raft (single group) where a log entry == serialized WriteBatch
Deterministic apply + restart safety
Snapshotting for fast catch-up

Sequel teaser (maybe)

Tables & Regions: table-ish encoding + scans + key-range routing (minimal, no rabbit holes)

Non-goals (by design)

To keep BeachDB small and finishable, these are intentionally out of scope for Season 1:

Production readiness, multi-year maintenance guarantees, or compatibility promises
Multi-writer concurrency in the engine (single-writer early on)
Background compaction early on (added only after invariants are rock-solid)
SQL, query planner, joins, secondary indexes
Full transactions / serializable isolation
Auto sharding, region split/merge, rebalancing, quorum reads, gossip/repair

Philosophy

Every chapter ends with evidence: a dump tool, a crash test, a benchmark, or a diagram.

See docs/principles.md to see how I'm keeping this project from turning into a second job :)

License

Apache 2.0 (see: LICENSE)

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
crash command Package main provides a controller/worker crash harness for exercising BeachDB durability under process crashes and injected engine fault points.	Package main provides a controller/worker crash harness for exercising BeachDB durability under process crashes and injected engine fault points.
manifest_dump command Package main provides the manifest_dump CLI tool for inspecting MANIFEST files.	Package main provides the manifest_dump CLI tool for inspecting MANIFEST files.
sst_dump command Package main provides the sst_dump CLI tool for inspecting SST files.	Package main provides the sst_dump CLI tool for inspecting SST files.
wal_dump command Package main provides the wal_dump CLI tool for inspecting WAL files.	Package main provides the wal_dump CLI tool for inspecting WAL files.
engine Package engine exposes the public BeachDB storage engine API.	Package engine exposes the public BeachDB storage engine API.
examples
engine/basic_usage command
engine/batch_operations command
engine/crash_recovery command
engine/flush_threshold command
engine/manifest command
engine/options command
internal
crashhook Package crashhook exposes internal-only crash and fault injection points used by the crash harness and engine tests.	Package crashhook exposes internal-only crash and fault injection points used by the crash harness and engine tests.
fs Package fs provides filesystem durability helpers shared by the engine, WAL, and manifest packages.	Package fs provides filesystem durability helpers shared by the engine, WAL, and manifest packages.
keys Package keys define the internal key type of beachdb	Package keys define the internal key type of beachdb
manifest Package manifest implements the MANIFEST log and CURRENT pointer that track BeachDB's on-disk state across restarts.	Package manifest implements the MANIFEST log and CURRENT pointer that track BeachDB's on-disk state across restarts.
memtable Package memtable implements BeachDB's in-memory write buffer using a skip list.	Package memtable implements BeachDB's in-memory write buffer using a skip list.
record Package record implements BeachDB's shared append-log record framing.	Package record implements BeachDB's shared append-log record framing.
sstable Package sstable implements the SSTable (Sorted String Table) format for BeachDB's LSM storage engine.	Package sstable implements the SSTable (Sorted String Table) format for BeachDB's LSM storage engine.
testutil Package testutil provides test helpers for BeachDB.	Package testutil provides test helpers for BeachDB.
util/checksum Package checksum provides CRC32C checksum utilities for data integrity verification.	Package checksum provides CRC32C checksum utilities for data integrity verification.
util/coding Package coding provides utilities for encoding and decoding binary data.	Package coding provides utilities for encoding and decoding binary data.
wal Package wal implements the Write-Ahead Log for durability and crash recovery.	Package wal implements the Write-Ahead Log for durability and crash recovery.