docstore

README

go-docstore

A tiny document store on SQLite for Go: collections of JSON documents addressed by string ids — "CRUD by file names". The mental model of a directory of JSON files, with what that approach never gives you: real multi-process safety, transactions, indexes and search.

Born inside mist-drive, which stored users as JSON files with file locks until the locking, atomic-rename and cache-invalidation code grew into an unlabeled, untested database. This is that code done right — on top of the most battle-tested storage engine in existence.

Features

• One file, many collections — pure-Go SQLite (modernc.org/sqlite, no cgo)
• CRUD by id, path-like ids welcome ("user-1/upload-42")
• Transactional Update (read-modify-write under the write lock) — no lost updates, in-process or across processes
• Indexed lookups via SQLite generated columns over JSON paths (unique and case-insensitive variants) — documents stay schemaless
• Find — search on JSON content (= != < > like), literal SQLite json_extract paths
• FindPath — friendlier search: index-free dotted paths (loginHistory.at, no $., no [0]); if the first segment is an array, EVERY element is searched, not just the first
• FindAll/FindAny — compound search, AND/OR over multiple (path, op, value) conditions; FindAllIndexed does the same AND but seeks each condition's generated column instead of scanning, erroring if any path isn't indexed
• Metadata for free — created_at/updated_at per document, Meta(id)
• Opt-in soft delete — Delete marks, reads filter, Restore/Purge
• Files bridge — ImportDir/ImportFile migrate a tree of JSON files into a collection (ids = relative paths); ExportDir writes them back
• No app-level cache needed — SQLite's page cache serves reads in microseconds and is never stale for a second process
• Optional debug logging — one record per operation via SetLogger
• CLI (godocstore) to browse, edit and search any go-docstore database
• Debug web UI (godocstore serve) — JSON-first browser/editor: collections nav, syntax-highlighted documents, inline editing (validated, transactional), soft-delete lifecycle, Find and raw SQL tabs. Loopback-only by default (no auth)

Installation

Library:

go get github.com/creativeyann17/go-docstore

CLI:

go install github.com/creativeyann17/go-docstore/cmd/godocstore@latest
# or grab a prebuilt binary from Releases

CLI usage

godocstore --db app.db create users uploads        # new empty db + collections
godocstore --db app.db ls                          # collections + counts
godocstore --db app.db ls users                    # document ids
godocstore --db app.db get users u1 --meta         # pretty JSON + timestamps
godocstore --db app.db put users u1 doc.json       # upsert (stdin when no file)
godocstore --db app.db edit users u1               # $EDITOR round-trip, validated
godocstore --db app.db find users login = yann     # search JSON content
godocstore --db app.db find users quota '>' 1000000 --ids
godocstore --db app.db find users loginHistory.ip = 1.2.3.4  # any array entry, not just [0]
godocstore --db app.db sql "SELECT id, json_extract(doc,'$.email') FROM c_users"
godocstore --db app.db import users ./legacy-json/ # migrate files → documents
godocstore --db app.db export users ./backup/      # documents → files
godocstore --db app.db rm users u1 --soft          # mark; restore/purge undo it
godocstore --db app.db serve                       # JSON-first debug web UI (loopback)

Library usage

store, _ := docstore.Open("app.db")
defer store.Close()

users, _ := store.Collection("users",
    docstore.WithUniqueIndex("login", "$.login"),
    docstore.WithIndex("email", "$.email", true), // NOCASE
)

users.Put("u1", User{Login: "yann"})              // upsert
var u User
users.Get("u1", &u)                                // read (fresh copy, always)
users.GetBy("login", "yann", &u)                   // indexed lookup

// The concurrency primitive: transactional read-modify-write.
users.Update("u1", func(raw []byte) ([]byte, error) {
    var cur User
    json.Unmarshal(raw, &cur)
    cur.Quota += 50
    return json.Marshal(cur)
})

docs, _ := users.Find("$.quota", ">", 100, 10)     // search (literal SQLite path)
docs, _ = users.FindPath("loginHistory.ip", "=", "1.2.3.4", 10) // array-aware: any entry
docs, _ = users.FindAll([]docstore.Condition{                   // AND of conditions
    {Path: "$.quota", Op: ">", Value: 100},
    {Path: "$.login", Op: "=", Value: "yann"},
}, 10)
meta, _ := users.Meta("u1")                        // created/updated/deleted

See examples/ for a runnable walkthrough.

Semantics worth knowing

• Put keeps the original created_at and revives a soft-deleted document; Insert fails with ErrExists on any conflict.
• Put/Insert/Update require valid JSON (including when you pass raw []byte); invalid input is rejected before write.
• Collection tables are named c_<name> with the name preserved (foo-bar and foo_bar are distinct). Older databases that stored - as _ are still opened via that legacy table.
• Soft delete + unique indexes don't mix well: a soft-deleted document still holds its unique values, so re-creating "the same" document conflicts until purged. Prefer hard delete (the default) on collections with unique indexes.
• Databases created by mist-drive's embedded ancestor (schema without metadata columns) are upgraded in place on first Collection() call.
• The store uses WAL: a live database is app.db + -wal + -shm. Don't cp it while an app is writing — use VACUUM INTO 'backup.db'.

Development

make test           # unit tests
make build          # bin/godocstore
make build-all      # cross-compiled dist/ + checksums
make install-hooks  # gofmt pre-commit hook

Releases: tag vX.Y.Z → CI runs tests, builds all platforms, publishes with notes extracted from CHANGELOG.md.

License

MIT

Documentation

Overview

Package docstore is a small document store on SQLite: collections of JSON documents addressed by string ids — "CRUD by file names", with the directory/file mental model (path-like ids such as "uid/upload" are fine) but real multi-process safety.

Design rules:

• Only stdlib + the pure-Go SQLite driver (modernc.org/sqlite, no cgo).
• No app-level cache: SQLite's page cache serves hot reads in microseconds and is never stale across processes.
• The concurrency primitive is Update (transactional read-modify-write, BEGIN IMMEDIATE) — it replaces file locks and per-entity mutexes and is correct across processes.
• Indexed lookups use SQLite generated columns over JSON paths, so documents stay schemaless while lookups stay O(log n).
• Every document carries created_at/updated_at metadata; soft deletion (deleted_at) is opt-in per collection via WithSoftDelete.

Index

• Variables
• type Collection
  • func (c *Collection) Count() (n int, err error)
  • func (c *Collection) CountDeleted() (n int, err error)
  • func (c *Collection) Delete(id string) (err error)
  • func (c *Collection) Each(fn func(id string, raw []byte) error) (err error)
  • func (c *Collection) EachDeleted(fn func(id string, raw []byte) error) (err error)
  • func (c *Collection) ExistsBy(col string, val any, exceptID string) (found bool, err error)
  • func (c *Collection) ExportDir(dir string) (n int, err error)
  • func (c *Collection) Find(jsonPath, op string, value any, limit int) (out []Doc, err error)
  • func (c *Collection) FindAll(conds []Condition, limit int) (out []Doc, err error)
  • func (c *Collection) FindAllIndexed(conds []Condition, limit int) (out []Doc, err error)
  • func (c *Collection) FindAny(conds []Condition, limit int) (out []Doc, err error)
  • func (c *Collection) FindIndexed(jsonPath, op string, value any, limit int) (out []Doc, err error)
  • func (c *Collection) FindPath(path, op string, value any, limit int) (out []Doc, err error)
  • func (c *Collection) Get(id string, out any) (err error)
  • func (c *Collection) GetBy(col string, val any, out any) (err error)
  • func (c *Collection) GetRaw(id string) (raw []byte, err error)
  • func (c *Collection) IDs() (out []string, err error)
  • func (c *Collection) ImportDir(dir string) (n int, err error)
  • func (c *Collection) ImportFile(path string) (err error)
  • func (c *Collection) Indexes() ([]IndexInfo, error)
  • func (c *Collection) Insert(id string, doc any) (err error)
  • func (c *Collection) Meta(id string) (m Meta, err error)
  • func (c *Collection) Purge(id string) (err error)
  • func (c *Collection) Put(id string, doc any) (err error)
  • func (c *Collection) Restore(id string) (err error)
  • func (c *Collection) Update(id string, fn func(raw []byte) ([]byte, error)) (err error)
• type Condition
• type Doc
• type IndexInfo
• type Meta
• type Option
  • func WithIndex(col, jsonPath string, nocase bool) Option
  • func WithSoftDelete() Option
  • func WithUniqueIndex(col, jsonPath string) Option
• type Store
  • func Open(path string) (*Store, error)
  • func (s *Store) Close() error
  • func (s *Store) Collection(name string, opts ...Option) (*Collection, error)
  • func (s *Store) Collections() ([]string, error)
  • func (s *Store) DB() *sql.DB
  • func (s *Store) SetLogger(l *slog.Logger)

Variables

var (
	ErrNotFound = errors.New("docstore: not found")
	ErrExists   = errors.New("docstore: already exists")
)

Types

type Collection

type Collection struct {
	// contains filtered or unexported fields
}

Collection is a named set of documents — think "directory".

func (*Collection) Count

func (c *Collection) Count() (n int, err error)

Count returns the number of (live) documents.

func (*Collection) CountDeleted

func (c *Collection) CountDeleted() (n int, err error)

CountDeleted returns the number of soft-deleted documents (deleted_at set). Works regardless of WithSoftDelete — it counts the column.

func (*Collection) Delete

func (c *Collection) Delete(id string) (err error)

Delete removes the document — hard by default, or marks deleted_at on WithSoftDelete collections. ErrNotFound when there is nothing (live) to delete.

func (*Collection) Each

func (c *Collection) Each(fn func(id string, raw []byte) error) (err error)

Each streams every (live) (id, raw document) pair, sorted by id. Returning an error from fn stops the iteration and propagates it.

func (*Collection) EachDeleted

func (c *Collection) EachDeleted(fn func(id string, raw []byte) error) (err error)

EachDeleted streams soft-deleted documents only.

func (*Collection) ExistsBy

func (c *Collection) ExistsBy(col string, val any, exceptID string) (found bool, err error)

ExistsBy reports whether any document other than exceptID has this value in the indexed column (pass "" to check all documents).

func (*Collection) ExportDir

func (c *Collection) ExportDir(dir string) (n int, err error)

ExportDir writes every (live) document to dir as <id>.json (pretty printed); path-like ids become subdirectories. Returns the number of exported documents. Ids that would escape dir are rejected.

func (*Collection) Find

func (c *Collection) Find(jsonPath, op string, value any, limit int) (out []Doc, err error)

Find returns documents whose json_extract(doc, jsonPath) matches value under op (one of = != < > like). value is used as-is: pass a number for numeric JSON fields, a string for text. limit <= 0 means no limit.

func (*Collection) FindAll

func (c *Collection) FindAll(conds []Condition, limit int) (out []Doc, err error)

FindAll returns documents matching every condition (AND), via json_extract per row like Find — a full scan even if some paths are indexed. Use FindAllIndexed when every condition's path is indexed and an index seek is wanted instead.

func (*Collection) FindAllIndexed

func (c *Collection) FindAllIndexed(conds []Condition, limit int) (out []Doc, err error)

FindAllIndexed is FindAll restricted to conditions whose paths all have a WithIndex/WithUniqueIndex on them — same real-index-seek technique as FindIndexed, applied per condition, so SQLite can seek each generated column instead of scanning with json_extract. Errors up front if any condition's path isn't indexed.

func (*Collection) FindAny

func (c *Collection) FindAny(conds []Condition, limit int) (out []Doc, err error)

FindAny returns documents matching at least one condition (OR), via json_extract per row like Find.

func (*Collection) FindIndexed

func (c *Collection) FindIndexed(jsonPath, op string, value any, limit int) (out []Doc, err error)

FindIndexed is Find restricted to a jsonPath that has a WithIndex/ WithUniqueIndex on it. Find's json_extract(doc, ?) is a per-row function call over a bound parameter — SQLite never matches that back to a generated column's index, even when jsonPath is the exact one an index was built on, so every Find is a full table scan regardless of what indexes the collection has. FindIndexed instead queries the underlying generated column by name (same technique GetBy/ExistsBy already use), which is a real index seek, and returns an error up front if jsonPath isn't indexed, so an accidentally-unindexed lookup fails immediately instead of quietly degrading into a scan. Use Find (or FindPath) when a full scan is genuinely intended.

func (*Collection) FindPath

func (c *Collection) FindPath(path, op string, value any, limit int) (out []Doc, err error)

FindPath is a friendlier variant of Find for simple, index-free dotted paths (e.g. "login", "address.city", "loginHistory.at" — no leading "$." and no array indices; a leading "$." is tolerated and stripped for muscle-memory compatibility).

The key difference from Find: if the FIRST path segment turns out to be an array, every element is searched — not just index 0. So "loginHistory.at" matches a document if ANY entry in its loginHistory array has that field equal to value, and a bare "tags" matches if value is a member of the tags array (array of scalars). A path whose first segment is not an array falls back to a plain field match, so "login" behaves exactly as expected.

Only the first segment is treated as a potential array boundary — deeper/multiple array hops need Find with a literal SQLite json_extract path (e.g. "$.a[2].b[0].c") instead.

func (*Collection) Get

func (c *Collection) Get(id string, out any) (err error)

Get unmarshals the document with this id into out.

func (*Collection) GetBy

func (c *Collection) GetBy(col string, val any, out any) (err error)

GetBy looks a document up through an indexed column (see WithIndex / WithUniqueIndex). With multiple matches the smallest id wins.

func (*Collection) GetRaw

func (c *Collection) GetRaw(id string) (raw []byte, err error)

GetRaw returns the raw JSON of the document with this id.

func (*Collection) IDs

func (c *Collection) IDs() (out []string, err error)

IDs lists every (live) document id, sorted.

func (*Collection) ImportDir

func (c *Collection) ImportDir(dir string) (n int, err error)

ImportDir walks dir recursively and stores every *.json file; ids are the slash-separated relative paths without the .json extension (so "users/id1.json" becomes id "users/id1"). Returns the number of imported documents; the first invalid file aborts with an error.

func (*Collection) ImportFile

func (c *Collection) ImportFile(path string) (err error)

ImportFile stores one JSON file as a document; the id is the file name without its .json extension. Invalid JSON is rejected.

func (*Collection) Indexes

func (c *Collection) Indexes() ([]IndexInfo, error)

Indexes lists the indexes present on the collection's table, read directly from the SQLite schema (sqlite_master) rather than from the Options the collection happens to have been opened with — so it reports the truth even when opened generically (e.g. by a browsing tool that doesn't know which WithIndex/WithUniqueIndex created them).

func (*Collection) Insert

func (c *Collection) Insert(id string, doc any) (err error)

Insert stores a NEW document; any conflict (id or unique index) returns ErrExists — including a conflict with a soft-deleted document, which still occupies its id.

func (*Collection) Meta

func (c *Collection) Meta(id string) (m Meta, err error)

Meta returns the document's metadata timestamps. Unlike Get, it also answers for soft-deleted documents (that's how you inspect them).

func (*Collection) Purge

func (c *Collection) Purge(id string) (err error)

Purge removes a document for real, regardless of soft-delete state.

func (*Collection) Put

func (c *Collection) Put(id string, doc any) (err error)

Put creates or replaces the document with this id, keeping its original created_at. Putting over a soft-deleted document revives it (deleted_at cleared). A unique-index conflict with a DIFFERENT document still returns ErrExists.

func (*Collection) Restore

func (c *Collection) Restore(id string) (err error)

Restore clears a soft-deleted document's deleted_at mark. Available on any collection (it acts on the column, not the option).

func (*Collection) Update

func (c *Collection) Update(id string, fn func(raw []byte) ([]byte, error)) (err error)

Update runs a transactional read-modify-write on one document: fn receives the current raw JSON and returns the replacement. The whole sequence holds the database write lock (BEGIN IMMEDIATE via the txlock DSN), so concurrent Updates — same process or another one — serialize instead of losing writes. fn returning an error aborts.

type Condition

type Condition struct {
	Path  string
	Op    string
	Value any
}

Condition is one clause of a compound query: json_extract(doc, Path) compared to Value under Op (same triple Find takes: = != < > like).

type Doc

type Doc struct {
	ID  string
	Raw []byte
}

Doc is one raw document, as returned by Find.

type IndexInfo

type IndexInfo struct {
	Name     string `json:"name"`     // the index's SQL name (ix_<table>_<col>)
	SQL      string `json:"sql"`      // the exact CREATE INDEX statement, as SQLite stored it
	Column   string `json:"column"`   // the generated column the index is on
	JSONPath string `json:"jsonPath"` // the json_extract path the column computes, e.g. "$.login"
	Unique   bool   `json:"unique"`
	NoCase   bool   `json:"nocase"`
}

IndexInfo describes one index found on a collection's table.

type Meta

type Meta struct {
	CreatedAt time.Time
	UpdatedAt time.Time
	DeletedAt *time.Time
}

Meta carries a document's metadata timestamps. DeletedAt is nil for live documents.

type Option

type Option func(*Collection)

Option configures a Collection at open time.

func WithIndex

func WithIndex(col, jsonPath string, nocase bool) Option

WithIndex adds a non-unique index; nocase makes lookups case-insensitive (à la strings.EqualFold).

func WithSoftDelete

func WithSoftDelete() Option

WithSoftDelete makes Delete mark documents (deleted_at) instead of removing them; reads filter marked documents out, Restore/Purge manage them. CAVEAT with unique indexes: a soft-deleted document still holds its unique values, so re-creating "the same" document conflicts until purged — prefer hard delete (the default) or no unique indexes on soft-delete collections.

func WithUniqueIndex

func WithUniqueIndex(col, jsonPath string) Option

WithUniqueIndex adds a UNIQUE index on json_extract(doc, jsonPath), exposed as a queryable column (GetBy/ExistsBy). Put/Insert of a conflicting document returns ErrExists.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is one SQLite database file holding any number of collections.

func Open

func Open(path string) (*Store, error)

Open opens (or creates) the database file. WAL journaling for concurrent readers + one writer, busy_timeout so competing writers queue instead of erroring, txlock=immediate so Update transactions take their write lock up front (no deferred-upgrade SQLITE_BUSY).

func (*Store) Close

func (s *Store) Close() error

func (*Store) Collection

func (s *Store) Collection(name string, opts ...Option) (*Collection, error)

Collection opens (creating if needed) a collection and applies its indexes. Idempotent — and it upgrades tables created by older versions of this package in place (adds missing metadata columns).

func (*Store) Collections

func (s *Store) Collections() ([]string, error)

Collections lists the collection names present in the database (tables matching the c_* naming scheme). Table names preserve the collection name (including '-'); older databases that stored '-' as '_' still list the underscore form.

func (*Store) DB

func (s *Store) DB() *sql.DB

DB exposes the underlying handle for consumers that need raw SQL (reports, ad-hoc queries). The document tables are named c_<collection>.

func (*Store) SetLogger

func (s *Store) SetLogger(l *slog.Logger)

SetLogger routes per-operation debug records (op, collection, id, duration, error) through the given logger. Nil (the default) means silent — callers opt in, typically with their app logger at debug level.