scriva

README

ScrivaDB

A lightweight, append-only, file-based document database — single binary, zero dependencies, human-readable storage.

---

Install

# Homebrew (via the tap)
brew install --cask srjn45/scriva/scriva

# Go
go install github.com/srjn45/scriva/cmd/scriva@latest

# Docker
docker run ghcr.io/srjn45/scriva

See Getting Started for Scoop, apt/rpm, and AUR channels.

---

Quick start

# 1. Start the server
scriva serve --data ./data --api-key dev-key

# 2. Insert a record
scriva-cli insert users '{"name":"alice","age":30}' --api-key dev-key

# 3. Query it
scriva-cli find users '{"field":"name","op":"eq","value":"alice"}' --api-key dev-key

Or use the interactive REPL:

scriva-cli repl --api-key dev-key

---

Run with Docker Compose

# Start the server (builds the image from the local Dockerfile on first run)
SCRIVA_API_KEY=dev-key docker compose up -d

# Insert a record against the REST gateway on :8080
curl -H "x-api-key: dev-key" \
  -d '{"data":{"name":"alice","age":30}}' \
  http://localhost:8080/v1/users/records

gRPC is exposed on :5433 and REST on :8080; data persists in the named scriva-data volume. See docker-compose.yml and the fully commented scriva.example.yaml for configuration.

---

What it is

ScrivaDB stores each collection as a set of NDJSON segment files — one JSON object per line, appended in order. There is no binary format, no external runtime, and no hidden state. You can inspect, backup, or migrate your data with any text editor or Unix tool.

Key properties:

• Append-only writes — inserts, updates, and deletes are always new lines; no in-place modification
• Configurable durability — choose none (OS flush), always (fsync per write), or interval (fsync on a timer) to trade throughput against crash-loss window
• End-to-end integrity — every segment entry carries a CRC32C checksum, so silent on-disk bit-rot is caught on read instead of returning wrong data
• Encryption at rest (embedded) — transparent field- or record-level AEAD encryption (XChaCha20-Poly1305) configured through the embedded façade: scriva.WithPassphrase/WithEncryptionKey/WithKeyProvider + WithCollectionEncryption(EncryptFields/EncryptRecord). Values are opaque on disk under a reserved marker while reads return plaintext; a wrong key/passphrase fails fast on open; key rotation and enable/disable migrate existing data lazily or in bulk via a re-encrypting compaction pass. Segments, backups, and replication carry ciphertext for free. See docs/encryption-at-rest.md
• Background compaction — a goroutine per collection merges and deduplicates sealed segments; operators can also force a synchronous pass on demand (scriva-cli compact)
• Online backup — scriva-cli backup streams a consistent gzip snapshot of the live database; restore is a plain tar xzf into a data directory
• Leader→follower replication — start a hot standby with --replicate-from <leader>: the follower bootstraps from a snapshot then tails the leader's committed writes (each tagged with a monotonic global LSN), applying them through the normal write path so its indexes match exactly. Async (bounded lag), resumes after a disconnect from its persisted applied-LSN with no gaps or duplicates, and exposes ReplicationStatus (leader LSN, per-follower lag).
• Read replicas — a follower serves reads (Find/FindById/FindByKey/Aggregate) from its applied state and refuses writes with a typed FAILED_PRECONDITION, so read traffic scales horizontally across followers. Bound staleness by diffing a follower's applied_lsn against the leader's leader_lsn
• Manual failover — after a leader loss, promote a caught-up follower to leader with the admin Promote RPC (scriva-cli promote): it stops replicating, lifts the read-only guard, and accepts writes. A lag guard refuses promoting a stale replica (override with --force); promotion is one-way and automatic election is out of scope. See the operator runbook
• In-memory index — O(1) lookup by id, persisted with a checksum for fast restarts
• Secondary indexes — per-field indexes for O(1) equality lookups and O(matches) range queries (gt/lt/…); automatically maintained and persisted
• Streaming queries — Find pushes limit/offset, a multi-field order_by_fields sort, and a keyset page_token cursor into the engine and streams results as it reads; a limited query is bounded by the page size, not the collection size, and honours client cancellation. Cursor pagination seeks past already-returned rows in O(page) — concatenated pages cover every row exactly once, no dupes or gaps. Optional field projection (--fields) returns only the requested fields (id/key/rev always included)
• Aggregations — Aggregate computes a count and numeric sum/avg/min/max over the same filter as Find, optionally grouped by a field; it streams one result per group and folds records in the engine (memory bounded by distinct groups, not rows), so clients count and total server-side instead of pulling the whole collection. CLI: aggregate --group-by --field --aggs count,sum,avg,min,max
• Transactions — optimistic multi-operation transactions via BeginTx / CommitTx / RollbackTx
• Keyed CRUD, upsert & CAS over the wire — caller-supplied string keys, insert-or-replace upsert, natural-key find/update/delete, and revision-checked compare-and-swap (update-if-rev) are now exposed over gRPC/REST (not just the embedded engine); every record carries a key and a monotonic rev, with duplicate/missing keys mapped to AlreadyExists/NotFound
• TTL / expiring records — per-record deadlines (--ttl / ttl_seconds on Insert & Update), a per-collection default (create-collection --default-ttl, persisted), and a server-wide default (--default-ttl); expired records are hidden from reads immediately and reclaimed by compaction
• gRPC + REST — dual API served from one binary; CLI uses the Unix socket when local
• OpenAPI spec — docs/openapi/scriva.swagger.json generated from the proto; generate clients for any language with openapi-generator
• Official client SDKs — idiomatic, hand-written libraries for 10 languages (Python, JavaScript/TypeScript, PHP, Java, Kotlin, Scala, Clojure, Ruby, Rust, C#/.NET) under clients/
• Scoped API keys — multiple named keys with read or read-write scope; a read-only key is refused on writes, and keys hot-reload on SIGHUP for rotation without a restart
• Optional TLS — TCP gRPC listener can be secured with a cert/key pair; CLI verifies via --tls-ca
• YAML config file — --config scriva.yaml with CLI flag overrides always winning
• Prometheus metrics — per-collection gauges, compaction histograms, gRPC request duration, and per-query rows-scanned at --metrics-addr
• Structured logging — leveled log/slog output (--log-level, --log-format json|text); one record per RPC with method, principal, duration, and status code
• Slow-query log — opt-in --slow-query-ms logs any Find over the threshold at WARN with filter shape, rows scanned vs returned, and whether an index was used, so unindexed hot queries surface from logs and metrics (off by default)
• Health & readiness — standard grpc.health.v1.Health service (SERVING → NOT_SERVING on graceful shutdown) plus HTTP /healthz (liveness) and /readyz (DB open + data dir writable) probes
• Backpressure & limits — opt-in --max-inflight in-flight ceiling and per-key --rate-limit token bucket shed load with RESOURCE_EXHAUSTED instead of unbounded resource growth; --max-concurrent-streams caps per-connection HTTP/2 streams (all off by default)
• Distributed tracing — opt-in OpenTelemetry spans to an OTLP collector (--otlp-endpoint, --otlp-sample-ratio); one span per RPC plus child engine.scan/engine.compaction spans, so a slow Find is traceable gateway → gRPC → engine (off by default; the embeddable engine gains no OTel dependency)
• Single binary — no JVM, no Python, no config files required to get started
• Web admin UI — browser-based collection and record manager at clients/web/ (React + Vite, talks to the REST gateway)

---

When to use it

ScrivaDB is the right tool when:

• You need persistence without standing up PostgreSQL or MongoDB
• Your data fits on one machine and you want human-readable files
• You're building CLI tools, local services, IoT daemons, or small web apps
• You want a simple HTTP/gRPC API you can call from any language

It is not the right tool for multi-node replication, complex joins, or datasets too large to compact on a single machine.

---

Embedding (use it as a Go library)

ScrivaDB's storage engine is a plain Go library, so you can skip the server entirely and run the database in-process — no gRPC, no network, no separate daemon. This is the right choice when your program is the only writer and you want ScrivaDB's durability and query model directly inside your binary.

go get github.com/srjn45/scriva/engine   # the storage engine
go get github.com/srjn45/scriva          # the ergonomic façade (recommended)

import "github.com/srjn45/scriva"

db, _ := scriva.Open("./data")            // embedded durability defaults (fsync ~1s)
defer db.Close()

sessions := db.MustCollection("sessions")
id, _, _ := sessions.InsertWithKey("sess-1", map[string]any{"status": "open"})
rec, _   := sessions.GetByKey("sess-1")   // caller-supplied string keys
_, _ = sessions.UpdateIfRev("sess-1", rec.Rev, map[string]any{"status": "closed"}) // CAS

Opt into transparent encryption at rest with a key option plus a per-collection policy — secrets are sealed on write and returned as plaintext on read, with nothing else in your code changing:

db, _ := scriva.Open("./data",
    scriva.WithPassphrase(os.Getenv("SCRIVA_PASSPHRASE")),
    scriva.WithCollectionEncryption("users", scriva.EncryptFields("password", "ssn")), // field-level
    scriva.WithCollectionEncryption("audit", scriva.EncryptRecord("id", "tenant")),     // record-level
)
users := db.MustCollection("users")
id, _, _ := users.Insert(map[string]any{"email": "a@b.com", "password": "hunter2"})
rec, _   := users.Get(id)          // rec.Data["password"] == "hunter2"; ciphertext on disk

A wrong passphrase fails fast on Open. Key rotation and enable/disable are runtime operations on the returned collection (RotateKey, SetEncryptionPolicy, MigrateNow, EncryptionStatus). See docs/encryption-at-rest.md for the full design.

The embedded surface includes caller-supplied string keys, per-record revisions with compare-and-swap, upsert, count/exists, secondary indexes, in-process Watch subscriptions, transparent encryption at rest, and a LoadJSONL bulk-import path. The engine pulls in no gRPC/protobuf/Prometheus/cobra/OpenTelemetry dependencies — a CI gate enforces that.

This is a distinct distribution channel: go get for embedding, while the standalone server ships via Homebrew/apt/GHCR and tagged binary releases. Both build from the same repo. See docs/embedding.md for the full API reference, durability modes, the Watch overflow contract, the versioning/stability policy, and a migration guide.

---

Client SDKs

Idiomatic, hand-written client libraries are available for ten languages. Each wraps the same gRPC API, takes the same connection config (host, port, api_key, optional TLS CA cert), and exposes every RPC including the streaming Find and Watch calls.

Language	Install	Reference
Python	pip install scriva	clients/python
JavaScript / TypeScript	npm i scriva	clients/js
PHP	composer require srjn45/scriva	clients/php
Java	io.github.srjn45:scriva-client (Maven Central)	clients/java
Kotlin	io.github.srjn45:scriva-client-kotlin:1.2.1 (Gradle, Maven Central)	clients/kotlin
Scala	"io.github.srjn45" %% "scriva-client-scala" % "1.2.1" (sbt, Maven Central)	clients/scala
Clojure	io.github.srjn45/scriva-client-clojure {:mvn/version "1.2.1"} (Clojars)	clients/clojure
Ruby	gem install scriva	clients/ruby
Rust	cargo add scriva	clients/rust
C# / .NET	dotnet add package Scriva.Client	clients/csharp

Prefer to generate your own? The checked-in OpenAPI spec covers every RPC — see Getting Started.

---

Documentation

Document	Description
Getting Started	Install, run, first queries, TLS, config file, secondary indexes, metrics, logging, health probes
Architecture	Storage model, write/read paths, compaction, secondary indexes, crash safety
Embedding	Use ScrivaDB as an in-process Go library: scriva/engine API, keyed ops, CAS, Watch, migration, versioning policy

---

Build from source

git clone https://github.com/srjn45/scriva
cd scriva
make build          # produces bin/scriva and bin/scriva-cli
make test           # run tests with race detector
make proto          # regenerate gRPC code (requires buf)

---

Contributing

Contributions are welcome — see CONTRIBUTING.md for how to build, test, and submit changes.

---

License

MIT — see LICENSE.

---

Rebuilt from FileDB PHP — original college project, now in Go.

Documentation

Overview

Package scriva is the embedded façade over the ScrivaDB storage engine. It is the ergonomic, zero-server entry point for programs that want to compile ScrivaDB in-process and host several collections — each with its own durability and compaction settings — from a single data directory.

It is a thin convenience layer over engine.DB: Open returns a handle, and Collection/MustCollection lazily open-or-create a named collection under per-collection options. Anything the façade does not expose is reachable via Engine.

Embedded durability default (OPS-1)

Unlike the raw engine — whose default is SyncModeNone (fastest, but a crash can lose recently acknowledged writes) — a DB opened through scriva.Open defaults every collection to SyncModeInterval at a 1s cadence. This trades a bounded (~1s) durability window for throughput: a crash can lose at most the last interval's writes, while the append-only, temp-then-rename segment format already rules out torn/partial records. It is the right default for a local, single-writer daemon that wants crash-safety without paying an fsync on every write.

Write paths that genuinely need per-write durability (a spend/ledger collection, say) can opt back into SyncModeAlways per collection with WithCollectionSyncMode(engine.SyncModeAlways) — an explicit escape hatch that fsyncs before every write is acknowledged. Every other engine default (segment size, compaction cadence, watch buffer) is left untouched unless overridden.

Index

• type CollectionOption
  • func WithCollectionCompactInterval(d time.Duration) CollectionOption
  • func WithCollectionSegmentMaxSize(n int64) CollectionOption
  • func WithCollectionSyncInterval(d time.Duration) CollectionOption
  • func WithCollectionSyncMode(m engine.SyncMode) CollectionOption
  • func WithCollectionWatchBufferSize(n int) CollectionOption
  • func WithMaxBytes(n uint64) CollectionOption
  • func WithMaxRecords(n uint64) CollectionOption
  • func WithUniqueIndex(fields ...string) CollectionOption
• type DB
  • func Open(dir string, opts ...Option) (*DB, error)
  • func (db *DB) Close() error
  • func (db *DB) Collection(name string, opts ...CollectionOption) (*engine.Collection, error)
  • func (db *DB) Engine() *engine.DB
  • func (db *DB) MustCollection(name string, opts ...CollectionOption) *engine.Collection
• type EncryptSpec
  • func EncryptFields(fields ...string) EncryptSpec
  • func EncryptRecord(indexFields ...string) EncryptSpec
  • func (s EncryptSpec) Policy() engine.EncryptionPolicy
• type Option
  • func WithCollectionEncryption(name string, spec EncryptSpec) Option
  • func WithCompactInterval(d time.Duration) Option
  • func WithEncryptionKey(key []byte) Option
  • func WithKeyProvider(p crypto.KeyProvider) Option
  • func WithPassphrase(passphrase string) Option
  • func WithSegmentMaxSize(n int64) Option
  • func WithSyncInterval(d time.Duration) Option
  • func WithSyncMode(m engine.SyncMode) Option
  • func WithWatchBufferSize(n int) Option

Types

type CollectionOption

type CollectionOption func(*collectionOptions)

CollectionOption overrides the DB-wide defaults for a single collection and/or declares unique indexes to ensure when the collection is opened.

func WithCollectionCompactInterval

func WithCollectionCompactInterval(d time.Duration) CollectionOption

WithCollectionCompactInterval overrides the compaction cadence for this collection.

func WithCollectionSegmentMaxSize

func WithCollectionSegmentMaxSize(n int64) CollectionOption

WithCollectionSegmentMaxSize overrides the segment rotation size for this collection.

func WithCollectionSyncInterval

func WithCollectionSyncInterval(d time.Duration) CollectionOption

WithCollectionSyncInterval overrides the flush cadence for this collection.

func WithCollectionSyncMode

func WithCollectionSyncMode(m engine.SyncMode) CollectionOption

WithCollectionSyncMode overrides the durability policy for this collection only. The headline use is WithCollectionSyncMode(engine.SyncModeAlways) for a write path (a spend/ledger collection) that needs an fsync on every write, while the rest of the store keeps the interval default.

func WithCollectionWatchBufferSize

func WithCollectionWatchBufferSize(n int) CollectionOption

WithCollectionWatchBufferSize overrides the Watch buffer size for this collection.

func WithMaxBytes

func WithMaxBytes(n uint64) CollectionOption

WithMaxBytes caps this collection's on-disk footprint at n bytes (S4, the summed size of its segment files): once the budget is reached, a write that would create a new record is refused with engine.ErrResourceExhausted. Like WithMaxRecords it gates only new-record creation, so a tenant at its limit can still update or delete to recover. Zero (the default) leaves it unlimited.

func WithMaxRecords

func WithMaxRecords(n uint64) CollectionOption

WithMaxRecords caps this collection at n live records (S4): an insert, keyed insert, inserting upsert, batch, or transaction that would create a record beyond the cap is refused with engine.ErrResourceExhausted, before anything is written. An in-place update or a delete is never refused. Zero (the default) leaves the record count unlimited.

func WithUniqueIndex

func WithUniqueIndex(fields ...string) CollectionOption

WithUniqueIndex ensures a unique secondary index on each named field when the collection is opened (via engine.Collection.EnsureUniqueIndex). Subsequent inserts or updates that would map a field's value to a different live record are rejected with engine.ErrDuplicateKey. Fields already indexed are left as they are.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is an embedded FileDB handle: a set of named collections rooted at one data directory, opened in-process with no server. It is safe for concurrent use.

func Open

func Open(dir string, opts ...Option) (*DB, error)

Open opens (or creates) an embedded database rooted at dir. Existing collections on disk are discovered automatically. Every collection defaults to SyncModeInterval at a 1s cadence (see the package doc for the OPS-1 rationale); pass Options to change the DB-wide defaults.

func (*DB) Close

func (db *DB) Close() error

Close closes every open collection and flushes their indexes to disk.

func (*DB) Collection

func (db *DB) Collection(name string, opts ...CollectionOption) (*engine.Collection, error)

Collection opens (or creates) the named collection, applying the DB-wide defaults overlaid with any CollectionOption. The first call for a given name wins: later calls return the same handle and ignore their options, so the per-collection config is fixed at first open. Use it once per collection at startup.

func (*DB) Engine

func (db *DB) Engine() *engine.DB

Engine returns the underlying engine.DB for operations the façade does not wrap (ListCollections, DropCollection, …). Collections opened directly on the returned handle bypass the façade's caching and per-collection option layer.

func (*DB) MustCollection

func (db *DB) MustCollection(name string, opts ...CollectionOption) *engine.Collection

MustCollection is Collection that panics on error. It is a convenience for package/struct initialization, where a store's fixed set of collections is opened once and a failure is fatal.

type EncryptSpec

type EncryptSpec struct {
	// contains filtered or unexported fields
}

EncryptSpec describes a collection's encryption policy. Build it with EncryptFields (field-level) or EncryptRecord (record-level) and pass it to WithCollectionEncryption.

func EncryptFields

func EncryptFields(fields ...string) EncryptSpec

EncryptFields seals a deny-list of named top-level fields, leaving every other field plaintext and queryable (field-level mode). At least one field is required; the reserved _key and secure_data fields cannot be named. Encrypted fields cannot be indexed, filtered, sorted, or aggregated on — they are opaque on disk.

func EncryptRecord

func EncryptRecord(indexFields ...string) EncryptSpec

EncryptRecord seals the whole record into a single opaque blob, keeping only the named index fields (and the reserved _key) plaintext and queryable (record-level mode). Name the fields you still need to filter or sort on; everything else moves into the sealed blob.

func (EncryptSpec) Policy

func (s EncryptSpec) Policy() engine.EncryptionPolicy

Policy returns the engine.EncryptionPolicy this spec describes. It lets the EncryptFields / EncryptRecord builders also drive a runtime policy change via Collection.SetEncryptionPolicy on a handle returned by DB.Collection, not just the open-time WithCollectionEncryption path.

type Option

type Option func(*engine.CollectionConfig)

Option configures the DB-wide defaults applied to every collection opened through the façade. Options mutate the base engine.CollectionConfig that is also handed to the engine when it pre-opens existing collections, so they take effect uniformly.

func WithCollectionEncryption

func WithCollectionEncryption(name string, spec EncryptSpec) Option

WithCollectionEncryption enables encryption for the named collection under spec. It is DB-wide: pass one per encrypted collection to Open. The policy takes effect when the collection is opened and is persisted in its meta.json, so it applies consistently across restarts. A key option (WithEncryptionKey / WithPassphrase / WithKeyProvider) must also be supplied, or opening the collection fails.

db, _ := scriva.Open("./data",
    scriva.WithPassphrase(os.Getenv("SCRIVA_PASSPHRASE")),
    scriva.WithCollectionEncryption("users", scriva.EncryptFields("password", "ssn")),
    scriva.WithCollectionEncryption("audit", scriva.EncryptRecord("id", "tenant")),
)

func WithCompactInterval

func WithCompactInterval(d time.Duration) Option

WithCompactInterval sets the background compaction cadence.

func WithEncryptionKey

func WithEncryptionKey(key []byte) Option

WithEncryptionKey configures a raw 32-byte encryption key for every collection that enables encryption. It is the entry point for apps that already manage key material (for example fetched from a KMS). The key must be exactly crypto.KeySize bytes; a wrong length fails at Open. For a human-supplied secret use WithPassphrase instead, and for an OS keychain / Vault / KMS integration use WithKeyProvider.

func WithKeyProvider

func WithKeyProvider(p crypto.KeyProvider) Option

WithKeyProvider wires in a custom crypto.KeyProvider — an OS keychain, Vault, a KMS, or a *crypto.Keyring you rotate yourself. It is the extension point behind which key rotation lives: Add a new key and SetCurrent to it on a keyring you own, then call Collection.RotateKey to seal new writes under it while old blobs stay readable by id.

func WithPassphrase

func WithPassphrase(passphrase string) Option

WithPassphrase derives the encryption key from a human-supplied passphrase via Argon2id. A random salt is minted the first time a passphrase-encrypted collection is created and persisted (non-secret) in that collection's meta.json; every subsequent Open re-derives the same key from the persisted salt, so the passphrase alone is enough to reopen. The passphrase itself is never written to disk. A wrong passphrase fails fast on Open with crypto.ErrWrongEncryptionKey.

func WithSegmentMaxSize

func WithSegmentMaxSize(n int64) Option

WithSegmentMaxSize sets the maximum active-segment size before rotation.

func WithSyncInterval

func WithSyncInterval(d time.Duration) Option

WithSyncInterval sets the flush cadence used under SyncModeInterval. The façade default is 1s.

func WithSyncMode

func WithSyncMode(m engine.SyncMode) Option

WithSyncMode overrides the default durability policy for every collection. The façade default is engine.SyncModeInterval; pass engine.SyncModeAlways for strict per-write fsync or engine.SyncModeNone to match the raw engine.

func WithWatchBufferSize

func WithWatchBufferSize(n int) Option

WithWatchBufferSize sets the per-subscriber Watch channel buffer.