ntee-db

README

ntee-db

A small, embedded, log-structured JSON key-value store written in pure Go, with prefix search and secondary indexes — usable from Go and Node.js.

An append-only JSONL file is the source of truth; an in-memory B-tree index serves fast lookups. No server, no separate process — the store runs inside your app, in the same spirit as lmdb or SQLite in embedded mode.

Highlights

• Log-structured & crash-safe — the data log is the WAL; torn writes are detected and truncated on boot, and the index is always rebuildable.
• Secondary indexes — string/number, multi-value, with exact / prefix / numeric-range queries, ±N first/last limits, and automatic per-value capping (MaxPerValue: keep only the newest N records per value).
• Flexible index schema — add, drop, or change indexes between opens with no migration step; dropped indexes are soft-deleted (data preserved, recoverable), and Reindex back-fills new indexes over existing records (see below).
• Prefix scans on the primary key; range delete by primary key for time-based pruning.
• Fast boot via a persisted index snapshot (hint) + log-tail replay.
• Disk-sized capacity, not RAM-sized — only keys + offsets are resident; values and large blobs stay on disk and are read on demand (hot reads served by the OS page cache). Unlike memory-bound stores such as Redis or memcached, the dataset is limited by disk, not RAM: a 1 GB-RAM device with a 40 GB disk can serve ~40 GB of data, as long as the key count fits the index budget (roughly key bytes + ~80 B of RAM per record). Degrades gracefully too — cold reads get slower instead of writes being refused.
• Atomic int64 counters — Incr/Decr with Redis-style semantics (missing key starts at 0, new value returned), plus bounded Topup/Take operators — Topup fills toward a max and returns how much didn't fit, Take subtracts only if the result stays above a floor — with the bound check and the update as one atomic operation, no read-then-write race. Counters are stored fixed-width and updated in place: a hot counter never grows the log and never creates compaction debt.
• Single-writer safety via kernel flock (releases on any process exit). Unix (macOS/Linux) only.
• Pure Go; the only dependency is tidwall/btree (MIT).

Packages

Directory	What it is
nteedb-core/	The store itself — a Go library (package nteedb). Full design & API docs in its README.
nteedb-js/	ntee-db, the Node.js binding — the core compiled as a c-shared library (source in nteedb-js/capi/), loaded via FFI, shipped with prebuilt binaries.
nteedb-server/	A standalone TCP server (redis/memcached-style daemon): text protocol, single-line JSON responses, parallel reads, optional auth. Protocol docs in its README.
nteedb-client-js/	ntee-db-client, a pure-JS TCP client for the server — every command, pipelining, zero runtime dependencies.

(The JS binding stays server-free — it embeds the core directly; the JS client is its networked counterpart, talking to nteedb-server over TCP.)

Running the server

go run ./nteedb-server -schema nteedb-server/schema.example.json

$ nc 127.0.0.1 6666
put call:1 {"kind":"request"}
{"ok":true,"result":true}
ix kind request
{"ok":true,"result":["call:1"]}

Using from Go

go get github.com/nickooan/ntee-db

import nteedb "github.com/nickooan/ntee-db/nteedb-core"

db, err := nteedb.Open(nteedb.Options{Dir: "/path/to/store"})
if err != nil { /* ... */ }
defer db.Close()

db.Put("input:GetOrders", []byte(`{"q": "orders"}`))
v, ok, err := db.Get("input:GetOrders")
keys, err := db.PrefixScan("input:Get")

See nteedb-core/README.md for the full story: design, key ordering, secondary indexes, MaxPerValue capping, schema migration, and all options.

Using from Node.js

npm install ntee-db

import { NteeDB } from "ntee-db";

const db = NteeDB.open("/path/to/store", {
  indexes: [{ name: "traceId", kind: "string" }],
});

db.put("call:1", { kind: "request" }, { traceId: "T1" });
const rec = await db.get("call:1"); // parsed JSON
await db.secIndex("traceId", "T1"); // ['call:1', ...]
db.close();

Prebuilt binaries ship for darwin-arm64, linux-amd64, linux-arm64 — no Go toolchain needed at install time. See nteedb-js/README.md for the API, benchmarks vs lmdb/better-sqlite3, and notes.

Evolving the index schema

ntee-db treats the index set as a declaration, not a migration: change Options.Indexes (Go), the indexes open option (JS), or schema.json (server) between opens and the store adopts the new set — never rejected:

• Dropped index → soft-dropped: it stops being maintained and queryable, but its data is preserved in the records (a tombstone entry in meta.json tracks it, and Compact deliberately keeps the data). Re-declare the index before a Reindex and its surviving data comes back.
• Added (or kind-changed) index → prospective: it covers records written from now on, but not history. DroppedIndexes() / ProspectiveIndexes() (server: dropped / prospective) report both states.
• Reindex() (server: reindex, admin) resolves everything at once: it rewrites every live record, re-running each index's extractor (Go Extract func / JS & server jsonPath) over the old values — so a newly added derived index gets back-filled across all existing data — and purges soft-dropped leftovers from records and meta.json. Reads stay live while it runs; writes wait.

The one asymmetry: only extractor-based indexes can be back-filled. An explicit-values index (values passed per write) stays prospective — its historical values were never recorded anywhere to recover. If you expect to add an index over a field later, storing that field in the record and using jsonPath/Extract keeps that door open.

Day-to-day writes never pay for any of this: extraction runs once at write time, the result is persisted in the record, and boots/compactions rebuild indexes from those persisted values without re-running extractors. Details in nteedb-core/README.md.

Building the server

The server is pure Go (no cgo), so every platform cross-compiles from any host in seconds — no Docker:

./build.sh                 # → bin/nteedb-server-{darwin-arm64,linux-amd64,linux-arm64}
./build.sh linux-arm64     # one target

Linux builds are fully static — scp and run, no runtime dependencies.

Development

go test -race ./...            # core + capi + server tests
./build.sh                     # server binaries (pure Go, cross-compiles)
nteedb-js/capi/build.sh        # JS native libs (macOS host + Linux via Docker)
cd nteedb-js && npm test       # Node binding tests
cd nteedb-client-js && npm test # TCP client tests (spawns a real server; needs Go)

License

Apache-2.0