docstore

README

go-docstore

A tiny document store on SQLite for Go: collections of JSON documents addressed by string ids — "CRUD by file names". The mental model of a directory of JSON files, with what that approach never gives you: real multi-process safety, transactions, indexes and search.

Born inside mist-drive, which stored users as JSON files with file locks until the locking, atomic-rename and cache-invalidation code grew into an unlabeled, untested database. This is that code done right — on top of the most battle-tested storage engine in existence.

Features

• One file, many collections — pure-Go SQLite (modernc.org/sqlite, no cgo)
• CRUD by id, path-like ids welcome ("user-1/upload-42")
• Transactional Update (read-modify-write under the write lock) — no lost updates, in-process or across processes
• Indexed lookups via SQLite generated columns over JSON paths (unique and case-insensitive variants) — documents stay schemaless
• Find — search on JSON content (= != < > like)
• Metadata for free — created_at/updated_at per document, Meta(id)
• Opt-in soft delete — Delete marks, reads filter, Restore/Purge
• Files bridge — ImportDir/ImportFile migrate a tree of JSON files into a collection (ids = relative paths); ExportDir writes them back
• No app-level cache needed — SQLite's page cache serves reads in microseconds and is never stale for a second process
• Optional debug logging — one record per operation via SetLogger
• CLI (godocstore) to browse, edit and search any go-docstore database

Installation

Library:

go get github.com/creativeyann17/go-docstore

CLI:

go install github.com/creativeyann17/go-docstore/cmd/godocstore@latest
# or grab a prebuilt binary from Releases

CLI usage

godocstore --db app.db create users uploads        # new empty db + collections
godocstore --db app.db ls                          # collections + counts
godocstore --db app.db ls users                    # document ids
godocstore --db app.db get users u1 --meta         # pretty JSON + timestamps
godocstore --db app.db put users u1 doc.json       # upsert (stdin when no file)
godocstore --db app.db edit users u1               # $EDITOR round-trip, validated
godocstore --db app.db find users '$.login' = yann # search JSON content
godocstore --db app.db find users '$.quota' '>' 1000000 --ids
godocstore --db app.db sql "SELECT id, json_extract(doc,'$.email') FROM c_users"
godocstore --db app.db import users ./legacy-json/ # migrate files → documents
godocstore --db app.db export users ./backup/      # documents → files
godocstore --db app.db rm users u1 --soft          # mark; restore/purge undo it

Library usage

store, _ := docstore.Open("app.db")
defer store.Close()

users, _ := store.Collection("users",
    docstore.WithUniqueIndex("login", "$.login"),
    docstore.WithIndex("email", "$.email", true), // NOCASE
)

users.Put("u1", User{Login: "yann"})              // upsert
var u User
users.Get("u1", &u)                                // read (fresh copy, always)
users.GetBy("login", "yann", &u)                   // indexed lookup

// The concurrency primitive: transactional read-modify-write.
users.Update("u1", func(raw []byte) ([]byte, error) {
    var cur User
    json.Unmarshal(raw, &cur)
    cur.Quota += 50
    return json.Marshal(cur)
})

docs, _ := users.Find("$.quota", ">", 100, 10)     // search
meta, _ := users.Meta("u1")                        // created/updated/deleted

See examples/ for a runnable walkthrough.

Semantics worth knowing

• Put keeps the original created_at and revives a soft-deleted document; Insert fails with ErrExists on any conflict.
• Soft delete + unique indexes don't mix well: a soft-deleted document still holds its unique values, so re-creating "the same" document conflicts until purged. Prefer hard delete (the default) on collections with unique indexes.
• Databases created by mist-drive's embedded ancestor (schema without metadata columns) are upgraded in place on first Collection() call.
• The store uses WAL: a live database is app.db + -wal + -shm. Don't cp it while an app is writing — use VACUUM INTO 'backup.db'.

Development

make test           # unit tests
make build          # bin/godocstore
make build-all      # cross-compiled dist/ + checksums
make install-hooks  # gofmt pre-commit hook

Releases: tag vX.Y.Z → CI runs tests, builds all platforms, publishes with notes extracted from CHANGELOG.md.

License

MIT

Documentation

Overview

Package docstore is a small document store on SQLite: collections of JSON documents addressed by string ids — "CRUD by file names", with the directory/file mental model (path-like ids such as "uid/upload" are fine) but real multi-process safety.

Design rules:

• Only stdlib + the pure-Go SQLite driver (modernc.org/sqlite, no cgo).
• No app-level cache: SQLite's page cache serves hot reads in microseconds and is never stale across processes.
• The concurrency primitive is Update (transactional read-modify-write, BEGIN IMMEDIATE) — it replaces file locks and per-entity mutexes and is correct across processes.
• Indexed lookups use SQLite generated columns over JSON paths, so documents stay schemaless while lookups stay O(log n).
• Every document carries created_at/updated_at metadata; soft deletion (deleted_at) is opt-in per collection via WithSoftDelete.

Index

• Variables
• type Collection
  • func (c *Collection) Count() (n int, err error)
  • func (c *Collection) Delete(id string) (err error)
  • func (c *Collection) Each(fn func(id string, raw []byte) error) (err error)
  • func (c *Collection) EachDeleted(fn func(id string, raw []byte) error) (err error)
  • func (c *Collection) ExistsBy(col string, val any, exceptID string) (found bool, err error)
  • func (c *Collection) ExportDir(dir string) (n int, err error)
  • func (c *Collection) Find(jsonPath, op string, value any, limit int) (out []Doc, err error)
  • func (c *Collection) Get(id string, out any) (err error)
  • func (c *Collection) GetBy(col string, val any, out any) (err error)
  • func (c *Collection) GetRaw(id string) (raw []byte, err error)
  • func (c *Collection) IDs() (out []string, err error)
  • func (c *Collection) ImportDir(dir string) (n int, err error)
  • func (c *Collection) ImportFile(path string) (err error)
  • func (c *Collection) Insert(id string, doc any) (err error)
  • func (c *Collection) Meta(id string) (m Meta, err error)
  • func (c *Collection) Purge(id string) (err error)
  • func (c *Collection) Put(id string, doc any) (err error)
  • func (c *Collection) Restore(id string) (err error)
  • func (c *Collection) Update(id string, fn func(raw []byte) ([]byte, error)) (err error)
• type Doc
• type Meta
• type Option
  • func WithIndex(col, jsonPath string, nocase bool) Option
  • func WithSoftDelete() Option
  • func WithUniqueIndex(col, jsonPath string) Option
• type Store
  • func Open(path string) (*Store, error)
  • func (s *Store) Close() error
  • func (s *Store) Collection(name string, opts ...Option) (*Collection, error)
  • func (s *Store) Collections() ([]string, error)
  • func (s *Store) DB() *sql.DB
  • func (s *Store) SetLogger(l *slog.Logger)

Variables

var (
	ErrNotFound = errors.New("docstore: not found")
	ErrExists   = errors.New("docstore: already exists")
)

Types

type Collection

type Collection struct {
	// contains filtered or unexported fields
}

Collection is a named set of documents — think "directory".

func (*Collection) Count

func (c *Collection) Count() (n int, err error)

Count returns the number of (live) documents.

func (*Collection) Delete

func (c *Collection) Delete(id string) (err error)

Delete removes the document — hard by default, or marks deleted_at on WithSoftDelete collections. ErrNotFound when there is nothing (live) to delete.

func (*Collection) Each

func (c *Collection) Each(fn func(id string, raw []byte) error) (err error)

Each streams every (live) (id, raw document) pair, sorted by id. Returning an error from fn stops the iteration and propagates it.

func (*Collection) EachDeleted

func (c *Collection) EachDeleted(fn func(id string, raw []byte) error) (err error)

EachDeleted streams soft-deleted documents only.

func (*Collection) ExistsBy

func (c *Collection) ExistsBy(col string, val any, exceptID string) (found bool, err error)

ExistsBy reports whether any document other than exceptID has this value in the indexed column (pass "" to check all documents).

func (*Collection) ExportDir

func (c *Collection) ExportDir(dir string) (n int, err error)

ExportDir writes every (live) document to dir as <id>.json (pretty printed); path-like ids become subdirectories. Returns the number of exported documents. Ids that would escape dir are rejected.

func (*Collection) Find

func (c *Collection) Find(jsonPath, op string, value any, limit int) (out []Doc, err error)

Find returns documents whose json_extract(doc, jsonPath) matches value under op (one of = != < > like). value is used as-is: pass a number for numeric JSON fields, a string for text. limit <= 0 means no limit.

func (*Collection) Get

func (c *Collection) Get(id string, out any) (err error)

Get unmarshals the document with this id into out.

func (*Collection) GetBy

func (c *Collection) GetBy(col string, val any, out any) (err error)

GetBy looks a document up through an indexed column (see WithIndex / WithUniqueIndex). With multiple matches the smallest id wins.

func (*Collection) GetRaw

func (c *Collection) GetRaw(id string) (raw []byte, err error)

GetRaw returns the raw JSON of the document with this id.

func (*Collection) IDs

func (c *Collection) IDs() (out []string, err error)

IDs lists every (live) document id, sorted.

func (*Collection) ImportDir

func (c *Collection) ImportDir(dir string) (n int, err error)

ImportDir walks dir recursively and stores every *.json file; ids are the slash-separated relative paths without the .json extension (so "users/id1.json" becomes id "users/id1"). Returns the number of imported documents; the first invalid file aborts with an error.

func (*Collection) ImportFile

func (c *Collection) ImportFile(path string) (err error)

ImportFile stores one JSON file as a document; the id is the file name without its .json extension. Invalid JSON is rejected.

func (*Collection) Insert

func (c *Collection) Insert(id string, doc any) (err error)

Insert stores a NEW document; any conflict (id or unique index) returns ErrExists — including a conflict with a soft-deleted document, which still occupies its id.

func (*Collection) Meta

func (c *Collection) Meta(id string) (m Meta, err error)

Meta returns the document's metadata timestamps. Unlike Get, it also answers for soft-deleted documents (that's how you inspect them).

func (*Collection) Purge

func (c *Collection) Purge(id string) (err error)

Purge removes a document for real, regardless of soft-delete state.

func (*Collection) Put

func (c *Collection) Put(id string, doc any) (err error)

Put creates or replaces the document with this id, keeping its original created_at. Putting over a soft-deleted document revives it (deleted_at cleared). A unique-index conflict with a DIFFERENT document still returns ErrExists.

func (*Collection) Restore

func (c *Collection) Restore(id string) (err error)

Restore clears a soft-deleted document's deleted_at mark. Available on any collection (it acts on the column, not the option).

func (*Collection) Update

func (c *Collection) Update(id string, fn func(raw []byte) ([]byte, error)) (err error)

Update runs a transactional read-modify-write on one document: fn receives the current raw JSON and returns the replacement. The whole sequence holds the database write lock (BEGIN IMMEDIATE via the txlock DSN), so concurrent Updates — same process or another one — serialize instead of losing writes. fn returning an error aborts.

type Doc

type Doc struct {
	ID  string
	Raw []byte
}

Doc is one raw document, as returned by Find.

type Meta

type Meta struct {
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt *time.Time
}

Meta carries a document's metadata timestamps. DeletedAt is nil for live documents.

type Option

type Option func(*Collection)

Option configures a Collection at open time.

func WithIndex

func WithIndex(col, jsonPath string, nocase bool) Option

WithIndex adds a non-unique index; nocase makes lookups case-insensitive (à la strings.EqualFold).

func WithSoftDelete

func WithSoftDelete() Option

WithSoftDelete makes Delete mark documents (deleted_at) instead of removing them; reads filter marked documents out, Restore/Purge manage them. CAVEAT with unique indexes: a soft-deleted document still holds its unique values, so re-creating "the same" document conflicts until purged — prefer hard delete (the default) or no unique indexes on soft-delete collections.

func WithUniqueIndex

func WithUniqueIndex(col, jsonPath string) Option

WithUniqueIndex adds a UNIQUE index on json_extract(doc, jsonPath), exposed as a queryable column (GetBy/ExistsBy). Put/Insert of a conflicting document returns ErrExists.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is one SQLite database file holding any number of collections.

func Open

func Open(path string) (*Store, error)

Open opens (or creates) the database file. WAL journaling for concurrent readers + one writer, busy_timeout so competing writers queue instead of erroring, txlock=immediate so Update transactions take their write lock up front (no deferred-upgrade SQLITE_BUSY).

func (*Store) Close

func (s *Store) Close() error

func (*Store) Collection

func (s *Store) Collection(name string, opts ...Option) (*Collection, error)

Collection opens (creating if needed) a collection and applies its indexes. Idempotent — and it upgrades tables created by older versions of this package in place (adds missing metadata columns).

func (*Store) Collections

func (s *Store) Collections() ([]string, error)

Collections lists the collection names present in the database (tables matching the c_* naming scheme). Note: '-' in a collection name is stored as '_' in the table name, so names round-trip with underscores.

func (*Store) DB

func (s *Store) DB() *sql.DB

DB exposes the underlying handle for consumers that need raw SQL (reports, ad-hoc queries). The document tables are named c_<collection>.

func (*Store) SetLogger

func (s *Store) SetLogger(l *slog.Logger)

SetLogger routes per-operation debug records (op, collection, id, duration, error) through the given logger. Nil (the default) means silent — callers opt in, typically with their app logger at debug level.