meldbase

package module
v0.1.0-alpha.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

README

Meldbase

Documents that stay live. Local by design.

Meldbase is an experimental, embedded reactive document database written in Go. It combines a typed document model, local durable storage, query planning, and live query subscriptions behind one coherent API. It is a new database—not a MongoDB protocol or behavior clone.

The current implementation contains a typed Go engine, crash-recoverable copy-on-write storage with a durable Commit Log, ordered compound B+Tree indexes, shared incremental reactive views and deltas, a Go/TypeScript query contract, and a secured HTTP/WebSocket reactive server. Open creates new databases with the single-file V2 engine and detects existing V1/V2 files without migration. OpenV1 remains available for deliberate legacy-format creation.

It is still early-stage: the V2 Meta negotiation envelope fails closed on unsupported revisions/features, and byte-exact cross-release fixtures now pin all current revision-3 PageTypes in a reachable multi-level business graph. Migration of existing V1 databases is explicit, and broader filesystem/platform durability evidence remains incomplete. B+Tree deletion uses local borrow/merge, while V2 physical nodes split by encoded bytes and expose structural write counters. New default databases use one main file; legacy V1 databases retain their .wal sidecar. V1 bounds that WAL with a default 64 MiB/10,000-commit automatic checkpoint policy; V2 checkpoints every COW commit directly. V2 also retains a bounded 10,000-commit / 256 MiB logical replay window by default, while active replay leases safely pin required history and expose retention pressure. All engines enforce canonical document/transaction resource limits before publication. V2 point transactions additionally admit tracked entries and retained base/current overlay bytes while the callback runs, preventing a read-heavy callback from growing unbounded before commit. V2 additionally has an 8 GiB default physical high-water quota; safe reusable pages are consumed first and crossing the quota rejects before file I/O without poisoning the database.

Installation

The Go module is published from:

go get github.com/crapthings/meldbase@latest

Before placing production-like data on a target volume, run the non-destructive capability probe in that directory:

go build -o /tmp/meldbase-qualification ./cmd/meld
/tmp/meldbase-qualification durability-check \
  --dir /path/to/database-volume \
  --source-revision "$(git rev-parse HEAD)" \
  --require-clean-source

It creates an isolated temporary directory, checks file and directory fsync, exclusive advisory-lock conflict and close-release behavior, atomic no-overwrite hard links, same-directory rename, and a real indexed V2 commit/reopen followed by offline full-graph verification. It prints schema-2 JSON and removes the probe directory. Passing proves those APIs work in the current mounted environment; it does not prove controller behavior during power loss. See filesystem qualification.

Release candidates use an explicitly built, clean-revision binary for the workload-independent probe and the concurrent storage soak:

go build -race -o /tmp/meldbase-qualification ./cmd/meld
/tmp/meldbase-qualification storage-soak \
  --dir /path/to/database-volume \
  --out storage-soak-receipt.json \
  --profile release --seconds 14400 --documents 10000 --reopens 12 \
  --source-revision "$(git rev-parse HEAD)" --require-clean-source

The schema-4 soak receipt binds the actual race-enabled binary revision and dirty flag to the exact target volume, then proves concurrent writes, snapshots, shadow-index catch-up, reclamation, repeated reopen and final offline semantics. Its four-hour release floor applies to measured concurrent worker time; reopen and verification overhead is reported separately and cannot satisfy the floor. The release writer runs flat out only until it proves one real optimistic reclamation conflict, then uses a hardware-independent cadence of one write every ten seconds. This keeps the duration/recovery qualification inside the normal V2 physical safety quota; it is not a storage-throughput benchmark. Shadow-index catch-up coalesces up to thirty seconds of ordered Commit Log work, then drains any larger backlog in bounded batches. This keeps the soak focused on realtime recovery behavior instead of synchronously mirroring every write with another COW commit. Sanitized 30-second stderr heartbeats expose phase progress and aggregate work without changing the canonical receipt or revealing database paths and values. meld qualification-check binds it to the schema-2 capability receipt. This is Level 3 evidence; real ENOSPC and power-cut qualification remains separate. The first Level 4 runner, meld destructive-process-check, now sends a real SIGKILL to a separate writer and produces an append-only-oracle receipt with old/new logical-state hashes, lock reacquisition and full offline verification. The Linux-only destructive-volume-check and destructive-enospc-check pair adds a token-gated, non-root, independently mounted disposable-volume runner that reaches every V2 publication boundary, fills the real filesystem through kernel fallocate until ENOSPC, kills the blocked writer and verifies recovery. destructive-qemu-reset captures the QMP block inventory, exact hard-reset command/response and host-originated RESET event. destructive-power-prepare and destructive-power-recover bind that proof to the exact boundary marker, Linux boot-ID change and untouched crash image. destructive-manifest-build re-verifies all retained artifacts and assembles the strict Level 4 record. The reference meld-power-redfish-adapter performs a target-bound ForceOff/observed-Off/On/observed-On cycle and embeds its redacted hardware log in the signed physical-controller proof. Level 4 storage qualification still requires actually executing the complete external power-cut boundary matrix described in filesystem qualification, and sets only storageQualified=true. A production claim requires the Level 5 gate, which additionally verifies the complete signed rollback-anchor phase chain and a separate multi-agent concurrent history, including clean release provenance for every controller and execution agent. Production Level 5 output is an independently signed, no-overwrite release envelope. qualification-packet-verify requires all original evidence and recomputes the qualification instead of trusting the archived boolean or its signature alone. Destructive evidence is rooted in a machine-generated content index for the exact secured artifact tree; added, removed, substituted or symlinked files fail Level 4 and Level 5 verification. A strict Linux environment capture additionally binds the target mount, kernel, block-device and cache-policy chain, external controller method and indexed operator authorization to the same campaign. Final verification uniquely locates every original destructive receipt from that index, reruns its retained-artifact checks and reconstructs the manifest rather than trusting its summary. The complete secured tree can be relocated without changing receipt bytes; indexed digests plus unique longest relative-path suffixes safely rebase old absolute artifact references. For the stronger VM-process loss case, destructive-qemu-process-kill verifies direct-I/O target binding, terminates QEMU with a real host SIGKILL, captures its wait status, and boots a new paused QEMU process before publishing evidence and continuing recovery. Power-matrix aggregation rejects repeated boot transitions and mixed controller methods, so reset and whole-process-loss evidence cannot be combined to satisfy coverage. destructive-corruption-check complements the crash matrices with a deterministic silent-bit-flip campaign over Meta, page header, payload, checksum and page-tail offsets. Every successful verification must still be a complete published generation with a full index semantic audit; the retained receipt can be independently replayed against the exact source database. The QEMU blkdebug EIO runner adds a distinct kernel-visible media-error case: it maps the seeded database's real ext4 sectors, injects write-only EIO, binds QMP BLOCK_IO_ERROR events to the exact config/image pair, and proves that the first write fails, subsequent writes remain poisoned, reads remain available and offline recovery preserves the previously published generation. The separate flush-state runner uses a durable guest ready → host QMP arm → guest transaction handshake, then injects one flush_to_disk/flush EIO on an ext4-backed virtio device. Its retained proof records both the exact injected rule and QMP's actual operation classification, and binds old-generation recovery to the seed and handshake receipts. This verifies handling of a reported persistence-barrier failure; it does not claim that successful flushes are truthful or cover stale reads. An explicit volatile-overlay negative control covers that distinction: it acknowledges a commit in a QEMU temporary layer, kills QEMU with SIGKILL, proves the durable base image never changed, and reboots without the layer. The harness must classify the resulting sequence rollback as unsafe. Public V2 opening supports a synchronous external rollback anchor; a stale but checksum-valid generation is rejected before the database file is modified:

anchor, err := meldbase.NewFileRollbackAnchorStore("/trusted/app.anchor")
if err != nil {
	return err
}
db, err := meldbase.OpenV2WithOptions("/data/app.meld", meldbase.V2Options{
	RollbackProtection: meldbase.V2RollbackProtection{
		AnchorStore:      anchor,
		InitializeAnchor: true, // audited first provisioning only
	},
})

Subsequent opens omit InitializeAnchor. Each logical commit or acknowledged maintenance generation first makes the database durable, then atomically advances and reads back the independent identity/sequence/generation anchor, and only then reports success. An anchor failure or the default 10-second anchor-operation timeout disables further writes. Failure is an ambiguous outcome—the anchor may already have advanced before its response was lost—so callers must not retry business logic as though the commit were absent. Custom authenticated remote quorum or TPM-backed monotonic stores implement the context-aware RollbackAnchorStore contract. The anchor must be on independently trusted storage; placing it beside the database cannot detect whole-device rollback.

integrations/anchorhttp provides a reference HTTPS/HMAC majority-read/majority-write store for one or an odd 2f+1 static set of independent nodes. Its fault tests cover stale minority state, one-node partition, duplicate advance, loss of quorum, authentication tampering, timeout, rejoin and rejection of a rolled-back database. It is crash-fault tolerant, not Byzantine-fault tolerant; safe dynamic membership remains separate work. Protocol v2 signs an automatically derived full-membership configuration and expected member identity, while every node directory is durably bound before serving. meld anchor-serve runs a standalone TLS 1.3 + mTLS member with private-file enforcement and graceful shutdown. The same guide documents the five-phase meld anchor-qualification workflow, whose Ed25519-signed receipt chain binds full-member probes, offline database verification and external network-controller evidence. It also documents history-agent/history-run, which execute a concurrent plan through independently signed, mTLS-authenticated and durably journaled agents, plus history-sign/history-verify, which bind the resulting controller ordinals, stable final member state and recomputed linearizability result into a separate signed receipt. The static safety model separately enumerates quorum publication/recovery behavior and documents the bounded TLA+ specification, concurrent scheduler, multi-process history checker and their fault-model boundaries.

Upgrade automation can inspect the newest checksum-valid Meta envelope without opening or locking the database:

go run ./cmd/meld inspect --db app.meld --require-compatible

The JSON reports V1/V2, revision, generation, commit sequence, required and optional feature bits, database identity, valid Meta slots and whether this reader is compatible. A checksum-valid future V2 revision is reported rather than misclassified as corruption; the compatibility gate exits with an error after emitting the JSON.

Before restore or archival promotion, perform the more expensive offline audit:

go run ./cmd/meld verify --db app.meld2 --timeout 10m

verify takes a shared advisory lock, walks the complete business graph protected by both valid Meta roots, and recomputes published secondary indexes against their canonical primary documents in both directions. Durable shadow builds are checked through their scan cursor or exact applied snapshot as appropriate. It also validates persistent FreeSpace separately and hashes every file byte. It emits a schema-versioned JSON receipt and never creates, truncates, repairs, reclaims, or advances the database. An active writer makes it fail with ErrDatabaseLocked.

The TypeScript packages currently live in this pnpm workspace and are not yet published to npm. Use the repository workspace for the developer preview; do not assume @meldbase/client or @meldbase/react is available from the public npm registry until an npm release is announced.

Go quick start

db, err := meldbase.Open("app.meld")
if err != nil { log.Fatal(err) }
defer db.Close()

users := db.Collection("users")
id, err := users.InsertOne(ctx, meldbase.Document{
  "name": meldbase.String("Ada"),
  "age":  meldbase.Int(30),
})

err = users.CreateIndex(ctx, "users_age", []meldbase.IndexField{
  {Field: "age", Order: 1},
}, meldbase.IndexOptions{})

// V2 and memory support one to four fields with independent directions.
err = users.CreateIndex(ctx, "users_name_age", []meldbase.IndexField{
  {Field: "name", Order: 1},
  {Field: "age", Order: -1},
}, meldbase.IndexOptions{Unique: true})

birthday, err := meldbase.CompileUpdate(meldbase.Update{
  "$inc": map[string]any{"age": 1},
})
if err != nil { log.Fatal(err) }

// Storage V2 atomically publishes point writes across collections. Meldbase
// never retries this callback when another commit wins first.
err = db.RunWriteTransaction(ctx, func(tx *meldbase.WriteTransaction) error {
  if err := tx.UpdateOne("users", id, birthday); err != nil {
    return err
  }
  _, err := tx.InsertOne("audit", meldbase.Document{
    "userId": meldbase.ID(id),
    "event":  meldbase.String("birthday"),
  })
  return err
})

Compound queries use a contiguous left prefix: equality fields followed by at most one range field. A missing first field omits a document; a missing suffix is represented internally so prefix queries remain complete without creating a unique-key conflict. Uniqueness applies only to complete tuples. Legacy V1 deliberately rejects compound and descending definitions.

RunWriteTransaction reads one immutable V2 snapshot, supports GetOne, Find, InsertOne, ReplaceOne, UpdateOne and DeleteOne, and publishes all effective changes under one commit token. Find uses the same compiled query syntax and sees earlier writes from its callback. It installs a durable collection snapshot fence, so any concurrent document or published-index change to that collection returns ErrWriteConflict rather than admitting a phantom. The first implementation intentionally uses a collection-wide conflict domain; it is serializable but may conflict for unrelated writes in the same collection. Candidate count and bytes are bounded by the transaction resource limits. A callback error or ErrWriteConflict publishes nothing; a no-op does not advance the sequence. Keep callbacks short: do not retain the transaction, call normal database methods, perform network I/O or create external side effects. Legacy V1 and the in-memory engine return ErrWriteTransactionUnsupported.

The TypeScript remote client also provides typed request/response RPC. HTTP is the default for work that does not need a realtime connection:

const receipt = await client.call("orders.checkout", [orderId, 2n]);

// Optional: multiplex the same envelope over an existing realtime socket.
const liveReceipt = await client.call("orders.checkout", [orderId, 2n], {
  transport: "realtime",
  signal: controller.signal,
});

// Caller-owned retry identity; Meldbase never retries RPC automatically.
const durableReceipt = await client.call("orders.checkout", [orderId, 2n], {
  idempotencyKey: crypto.randomUUID(),
});

RPC reuses Meldbase wire values, so bigint, Date and binary values do not lose type information. The public Go server package requires both explicit method registration and a separate RPC authorizer; arbitrary server error text is never sent to clients. Socket calls are never automatically retried after disconnect. An explicit V2-backed server store can durably claim an idempotency key and replay its terminal result; an interrupted claim returns outcome-unknown rather than rerunning application code. Go methods registered through RPCTransactionalMethods can additionally publish supported point writes and the successful result in one V2 commit; exact point-read-set contention returns a durable conflict without rerunning the method, while disjoint commits may proceed. See docs/client-protocol.md.

Trusted Node.js business methods can run out of process through @meldbase/server. A separately authenticated worker hub dynamically resolves ordinary/transactional methods and data-only query publications while Go retains authorization intersection, typed-value validation, transaction commit, field projection and reactive publication. Worker credentials are sent in control headers rather than URLs, and the control handler is intended for a private listener. See docs/server-js-sdk.md.

Run the worker example locally in two terminals:

export MELDBASE_WORKER_TOKEN=development-worker-token-0123456789abcdef
go run ./cmd/meld serve --db /tmp/meldbase-worker-demo.meld2 \
  --addr 127.0.0.1:8080 --dev-no-auth --worker-addr 127.0.0.1:9092 \
  --worker-publications orders

export MELDBASE_WORKER_TOKEN=development-worker-token-0123456789abcdef
pnpm --filter @meldbase/example-server-worker start

Then invoke its transactional method:

curl -sS http://127.0.0.1:8080/v1/rpc \
  -H 'content-type: application/json' \
  --data '{"v":1,"type":"call","requestId":"demo-1","idempotencyKey":"demo-order-000000000001","method":"orders.create","arguments":[{"t":"string","v":"first order"}]}'

The example also owns visibility for orders: it exposes only rows whose owner matches the authenticated subject and only its declared result fields. Go must predeclare orders through --worker-publications; if the worker is offline, queries to that managed collection fail closed.

V1-to-V2 migration is explicit and never overwrites its destination:

format, err := meldbase.DetectStorageFormat("app.meld")
if err != nil { log.Fatal(err) }
if format == meldbase.StorageFormatV1 {
  if err := db.MigrateToV2(ctx, "app-v2.meld"); err != nil {
    log.Fatal(err)
  }
}

Migration preserves empty collections, document insertion order and index definitions, but intentionally assigns a new database identity. Existing V1 realtime resume tokens therefore resynchronize instead of crossing formats.

V2 can compact live state into a separately verified file without overwriting the source. It pins one source snapshot briefly, then copies and verifies it without blocking later commits; writes that finish after that snapshot are not included in the compacted file:

if err := db.CompactToV2(ctx, "app-compacted.meld2"); err != nil {
  log.Fatal(err)
}

The compacted file also has a new database identity. Lazy V2 COLLSCAN cursors release their snapshot automatically on exhaustion, limit, error or context cancellation; callers that stop early should call cursor.Close().

V2 can also create an exact, checksummed physical restore artifact. Unlike compaction, backup deliberately preserves the database identity, Meta generation, Commit Log and physical history:

result, err := db.BackupV2(ctx, "app-backup.meld2")
if err != nil {
  log.Fatal(err)
}
log.Printf("backup sequence=%d sha256=%s", result.CommitSequence, result.SHA256)

For an offline database, the CLI performs the same validation and emits a schema-versioned JSON receipt:

go run ./cmd/meld backup --db app.meld2 --out app-backup.meld2 --timeout 10m
go run ./cmd/meld inspect --db app-backup.meld2 --require-compatible
go run ./cmd/meld verify --db app-backup.meld2 --timeout 10m

Restore requires the JSON receipt emitted by backup and always writes a new, previously absent database file:

go run ./cmd/meld backup --db app.meld2 --out app-backup.meld2 --timeout 10m > app-backup.receipt.json
go run ./cmd/meld restore --in app-backup.meld2 --receipt app-backup.receipt.json --out app-restored.meld2 --timeout 10m
go run ./cmd/meld verify --db app-restored.meld2 --timeout 10m

For the complete single-node data-directory check, health probes, dashboard, backup/restore drill and upgrade runbook, see single-node deployment and recovery.

The destination must not exist. The library blocks source writes for the copy duration while allowing readers; the CLI must acquire the database's exclusive process lock, so it is intended for an offline source. A physical backup is a restore artifact, not an independent writable clone: retire the original before starting the restored file. Use CompactToV2 when an independent database with a new identity and history is required.

V2 reclamation can run as an explicit low-pause maintenance loop. It is off by default; online scans do not hold the writer lock and discard their result if a commit changes the audited generation:

maintenance, err := db.StartV2Maintenance(ctx, meldbase.V2MaintenanceOptions{
  Interval:    5 * time.Minute,
  Timeout:     time.Minute,
  MaxAttempts: 2,
})
if err != nil {
  log.Fatal(err)
}
defer maintenance.Stop()

Runs are serial, deadline-bounded and stop automatically when the DB closes. They default to memory-only pool installation so the final writer pause is O(1). Set PersistFreeSpace: true only when restart acceleration is worth an explicit physical maintenance/fsync step.

Deliberate legacy V1 deployments can tune or disable its synchronous automatic checkpoint policy:

db, err := meldbase.OpenV1WithOptions("legacy.meld", meldbase.V1Options{
  Checkpoint: meldbase.V1CheckpointPolicy{
    MaxWALBytes:   128 << 20,
    MaxWALCommits: 20_000,
  },
})

Either threshold triggers. The triggering business commit is already durable; checkpoint maintenance does not advance the logical commit sequence.

Startup recovery is explicit and auditable. Normal Open performs only bounded automatic recovery and freezes the result in db.RecoveryReport(). Deployments that require an operator/offline verifier to approve every recovery can reject before any crash tail is truncated or WAL is replayed:

db, err := meldbase.OpenWithOptions("app.meld2", meldbase.OpenOptions{
  Recovery: meldbase.RecoveryRequireClean,
})
if errors.Is(err, meldbase.ErrRecoveryRequired) {
  // Keep the file untouched; run meld inspect/verify and follow site policy.
}

There is no online API for clearing durability fail-stop.

For a production process that prefers a slower startup to serving from a structurally corrupt deep V2 page, opt into the protected-graph audit. It runs before automatic crash-tail removal and does not replace the offline semantic index verifier:

db, err := meldbase.OpenV2WithOptions("app.meld2", meldbase.V2Options{
  RequireGraphAudit: true,
})

Write admission and V2 replay history are configured at open and remain immutable for that handle. Zero fields select the production defaults:

db, err := meldbase.OpenWithOptions("app.meld2", meldbase.OpenOptions{
  V2CommitRetention: meldbase.V2CommitRetentionPolicy{
    MaxCommits: 25_000,
    MaxBytes:   512 << 20,
  },
  // A full replay-consumer buffer is terminated after this interval so it
  // cannot indefinitely pin retained history. Zero selects five seconds.
  V2ReplayDeliveryTimeout: 5 * time.Second,
  V2StorageLimits: meldbase.V2StorageLimits{MaxFileBytes: 16 << 30},
  ResourceLimits: meldbase.ResourceLimits{
    MaxDocumentBytes:      8 << 20,
    MaxTransactionBytes:   32 << 20,
    MaxTransactionChanges: 5_000,
    MaxIndexBuildEntries:  500_000,
    MaxIndexBuildBytes:    128 << 20,
    MaxReactiveViewDocuments: 10_000,
    MaxReactiveViewBytes:     64 << 20,
  },
})

Resource bytes are deterministic typed binary sizes, not JSON or Go heap size. Index-build bytes count each encoded scalar key plus its 8-byte insertion position and 16-byte document ID. Oversized writes and index builds fail atomically with ErrResourceLimit; the defaults cap one build at 1,000,000 entries and 256 MiB. Online index creation leaves the commit sequence and durable bytes unchanged on rejection. The compatibility CreateIndex path uses a pinned snapshot without holding the database writer mutex and retries bounded snapshot conflicts. Storage V2 also supports durable, crash-resumable construction for write-heavy collections:

id, err := users.StartIndexBuild(ctx, "users_email_created",
  []meldbase.IndexField{{Field: "email", Order: 1}, {Field: "createdAt", Order: -1}},
  meldbase.IndexOptions{Unique: true})
err = db.ResumeIndexBuild(ctx, id) // safe after cancellation or reopen

scheduler, err := db.StartIndexBuildScheduler(ctx,
  meldbase.IndexBuildSchedulerOptions{
    PollInterval: time.Second, RunTimeout: 250 * time.Millisecond,
    MaxConcurrency: 1, RunImmediately: true,
  })
defer scheduler.Stop()

The private shadow catches up through retained commits and becomes visible only through one atomic catalog publication. db.IndexBuilds() exposes progress and db.AbortIndexBuild(ctx, id) releases it explicitly. Admin JSON, the embedded dashboard, Prometheus and OpenTelemetry report aggregate phase/size metrics without collection or index-name labels. Logical compaction refuses to discard an unfinished durable build; finish or abort it first. Each new catch-up generation also protects the exact applied CatalogRoot, so offline verification can prove the private tree against its watermark even after older Commit Log entries are pruned. The scheduler is default-off, time-sliced, limited to one instance per database, and persists terminal failure reasons rather than retrying them indefinitely.

An offline/local operator can manage the same durable records without writing a Go program. Every successful command emits a schema-versioned JSON receipt and acquires the normal exclusive database lock:

go run ./cmd/meld index-build start \
  --db app.meld2 --collection users --name users_email_created \
  --field email:1 --field createdAt:-1 --unique
go run ./cmd/meld index-build list --db app.meld2
go run ./cmd/meld index-build resume --db app.meld2 --id <build-id> --timeout 10m
go run ./cmd/meld index-build abort --db app.meld2 --id <build-id>

resume never hides a uniqueness or resource failure: it exits unsuccessfully, and list can inspect the still-private task before an explicit abort. These are local file-management commands, not unauthenticated HTTP administration routes. Compaction inherits the source V2 quota. Use CompactToV2WithOptions or MigrateToV2WithOptions with V2DestinationOptions when a rewritten file needs a different quota or index-build budget; an undersized destination is never published.

TypeScript SDK

The same compiled, data-only query is used by the local cache and sent to the server. Native JS applications can use a local reactive collection directly:

import { LocalCollection } from "@meldbase/client"

const todos = new LocalCollection([
  { _id: "one", title: "Learn Meldbase", done: false }
])

const open = todos.find({ done: false }, {
  sort: [{ path: "title", direction: 1 }]
})

const stop = open.subscribe(snapshot => render(snapshot))

Remote queries use HTTP for fetch and a ticket-authenticated WebSocket for their ongoing state:

import { MeldbaseClient } from "@meldbase/client"

const db = new MeldbaseClient({
  baseUrl: "https://data.example.com",
  accessToken: () => auth.currentAccessToken(),
  // Enable after every server in a rolling deployment advertises its fixed
  // realtime/RPC capability descriptor.
  requireRealtimeProtocol: true
})

const query = db.collection("todos").find({ done: false })
const stop = query.subscribe(render, {
  onStatus: status => showSyncState(status.state)
})

const created = await db.collection("todos").insertOne({
  title: "Build something live",
  done: false
})

await db.collection("todos").updateOne(
  { _id: created._id },
  { $set: { done: true } }
)

React uses a thin useSyncExternalStore adapter over that same query object; it does not introduce a second query language:

import { useMemo } from "react"
import { useLiveQuery } from "@meldbase/react"

function OpenTodos({ db }: { db: MeldbaseClient }) {
  const query = useMemo(
    () => db.collection("todos").find({ done: false }),
    [db]
  )
  const { documents, status, error } = useLiveQuery(query)
  // Keep the query object stable; updates arrive over its WebSocket subscription.
  return <TodoList todos={documents} syncState={status} error={error} />
}

See docs/client-protocol.md for the realtime and security model.

A complete browser example lives in examples/realtime-todos. Run the development server above, then:

pnpm --filter @meldbase/example-realtime-todos dev

The example performs real HTTP mutations and WebSocket snapshots through the React adapter. Open it twice to observe the same query update in both views.

Run the end-to-end demo

The demo performs durable insert/update, creates and uses an index, observes a reactive query, closes the database, and proves the data after reopen:

go run ./cmd/meld demo

Run the HTTP/WebSocket server locally only with the explicit development-auth switch:

go run ./cmd/meld serve \
  --db ./app.meld \
  --addr :8080 \
  --dev-no-auth

--dev-no-auth grants every request full access and is intentionally required; it is not a production authentication mode. A production embedding supplies the server Authenticator and Authorizer implementations itself.

The development server can exercise the same fail-closed V2 anchor lifecycle. On the first audited provisioning only, use --rollback-anchor-init; omit it on every subsequent start:

go run ./cmd/meld serve \
  --db /data/app.meld \
  --rollback-anchor /trusted/app.anchor \
  --rollback-anchor-init \
  --rollback-anchor-timeout 10s \
  --dev-no-auth

/trusted must already exist and, for actual rollback protection, must not be on the same failure/rollback domain as /data.

The CLI can also connect to the independently deployed TLS 1.3/mTLS quorum with the complete --rollback-anchor-cluster, repeated --rollback-anchor-replica member=https://endpoint, name, HMAC key-file, CA and client-certificate flags. See the rollback-anchor service deployment guide.

To experience the separately secured embedded observability panel:

export MELDBASE_ADMIN_TOKEN='replace-with-at-least-32-random-bytes'
go run ./cmd/meld serve \
  --db ./app.meld \
  --addr :8080 \
  --dev-no-auth \
  --admin-addr 127.0.0.1:9091 \
  --admin-diagnostics \
  --admin-metrics

Then open http://127.0.0.1:9091/ and paste the token. The panel receives a fixed-history snapshot followed by an isolated SSE stream; it is not backed by a user collection or the business reactive pipeline. Its health strip separates database, durability, storage, realtime, telemetry and optional transport state; fixed explanations identify fail-stop writes, queue pressure and recent fallback events without exposing business data. It also shows Commit Log retention pressure and configured/rejected write resource budgets. Use --admin-diagnostics-all only for short sessions that need every query/commit; the default diagnostic mode retains bounded slow and failed operations. The storage panel also exposes physical generation, rollback protection, anchor sequence/generation lag, failures, maximum latency and configured timeout. Prometheus can scrape the separately authenticated GET /metrics endpoint when --admin-metrics is enabled.

Applications already using OpenTelemetry can register the fixed aggregate schema through integrations/otel. The adapter consumes the same sampler and requires an application-owned MeterProvider; Meldbase does not construct an OTel SDK or exporter. Short CPU profiles, heap profiles and Go runtime traces are available through admin.RuntimeProfiler with explicit duration/byte limits and no default HTTP endpoint. See observability.

The implemented transport endpoints are:

GET  /livez
GET  /readyz
GET  /health  (readiness-compatible alias)
POST /v1/collections/{collection}/query
POST /v1/collections/{collection}/documents
POST /v1/collections/{collection}/mutations
POST /v1/realtime/tickets
GET  /v1/realtime

/livez proves only that the Go handler can respond. /readyz and /health require the database to be both readable and writable; a closed database or a fail-stop durability error returns 503. Probe responses contain only a version, fixed status and readable/writable booleans—never paths or error text. This keeps liveness available during diagnosis without routing new application traffic to a database that can no longer commit.

HTTP queries carry the same versioned, data-only AST used by the SDK. With the development server above:

curl -X POST http://localhost:8080/v1/collections/todos/query \
  -H 'Content-Type: application/json' \
  --data '{
    "version": 1,
    "query": {
      "version": 1,
      "where": {"op":"compare","cmp":"eq","path":"done","value":{"t":"bool","v":false}},
      "sort": [{"path":"title","direction":1}]
    }
  }'

Browser realtime authentication is two-step: obtain a short-lived, single-use ticket over authenticated HTTP, then send it in the first WebSocket message. Credentials never appear in the WebSocket URL. The core V1 exchange is:

{"v":1,"type":"authenticate","ticket":"<single-use-ticket>"}
{"v":1,"type":"subscribe","requestId":"open-todos","collection":"todos","query":{"version":1,"where":{"op":"compare","cmp":"eq","path":"done","value":{"t":"bool","v":false}}}}
{"v":1,"type":"snapshot","requestId":"open-todos","subscriptionId":"<id>","token":"<signed-resume-token>","documents":[]}
{"v":1,"type":"unsubscribe","subscriptionId":"<id>"}

See docs/client-protocol.md for reconnect, resync_required, limits, origin checks, and row/field authorization.

Status

Early-stage and not suitable for production data. See docs/architecture.md and docs/roadmap.md. The low-cost metrics, bounded admin sampler and secured realtime stats stream are in docs/observability.md. The first-stage requirement-to-evidence map is in docs/mvp-audit.md.

Supported query operators are $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists, $and, $or, and $not. Meldbase defines these semantics itself and does not promise MongoDB compatibility.

Development

go test ./...
go test -race ./...
go vet ./...
go run ./cmd/meld demo
pnpm check
pnpm test
pnpm build:example

Contributions should follow CONTRIBUTING.md. Security reports must use the private process described in SECURITY.md, not a public issue. Maintainer release gates are documented in docs/releasing.md.

License

Licensed under the Apache License 2.0.

Documentation

Index

Constants

View Source 
const (
	IndexBuildPhaseScan    IndexBuildPhase = "scan"
	IndexBuildPhaseCatchUp IndexBuildPhase = "catch_up"
	IndexBuildPhaseReady   IndexBuildPhase = "ready"
	IndexBuildPhaseFailed  IndexBuildPhase = "failed"

	IndexBuildFailureNone           IndexBuildFailure = ""
	IndexBuildFailureUniqueConflict IndexBuildFailure = "unique_conflict"
	IndexBuildFailureResourceLimit  IndexBuildFailure = "resource_limit"
	IndexBuildFailureHistoryLost    IndexBuildFailure = "history_lost"
	IndexBuildFailureCanceled       IndexBuildFailure = "canceled"
	IndexBuildFailureInvalidIndex   IndexBuildFailure = "invalid_index"
)
View Source 
const (
	DefaultV2CommitCoordinatorMaxBatch   = 32
	DefaultV2CommitCoordinatorMaxPending = 1024
)
View Source 
const (
	V2PageSize            uint64 = 16 << 10
	DefaultV2MaxFileBytes uint64 = 8 << 30
)
View Source 
const (
	DefaultV2CommitRetentionMaxCommits uint64 = 10_000
	DefaultV2CommitRetentionMaxBytes   uint64 = 256 << 20
	// DefaultV2ReplayDeliveryTimeout bounds how long a replay source can wait
	// for a full caller buffer before it releases the retained-history lease.
	DefaultV2ReplayDeliveryTimeout = 5 * time.Second
)
View Source 
const (
	ReplicationHelloFrame  = "hello"
	ReplicationBatchFrame  = "batch"
	ReplicationAckFrame    = "ack"
	ReplicationResyncFrame = "resync_required"
)
View Source 
const (
	DefaultMaxDocumentBytes      uint64 = 16 << 20
	DefaultMaxTransactionBytes   uint64 = 64 << 20
	DefaultMaxTransactionChanges uint64 = 10_000
	DefaultMaxIndexBuildEntries  uint64 = 1_000_000
	DefaultMaxIndexBuildBytes    uint64 = 256 << 20
	// Reactive views retain matching document versions for incremental ordering
	// and updates, not merely the page currently emitted to a subscriber.
	DefaultMaxReactiveViewDocuments uint64 = 10_000
	DefaultMaxReactiveViewBytes     uint64 = 64 << 20
)
View Source 
const (
	DefaultReplicationMaxFrameBytes = 96 << 20
)
View Source 
const DefaultRollbackAnchorOperationTimeout = 10 * time.Second

DefaultRollbackAnchorOperationTimeout prevents a failed remote trust service from indefinitely holding database publication acknowledgement.

View Source 
const DefaultV2CommitCoordinatorMaxDelay = time.Millisecond
View Source 
const ReplicationProtocolVersion = 1

ReplicationProtocolVersion is intentionally separate from the browser realtime protocol. It transports durable database positions between trusted servers, not end-user query subscriptions.

Variables

View Source 
var (
	ErrClosed                            = errors.New("meldbase: database is closed")
	ErrInvalidDocument                   = errors.New("meldbase: invalid document")
	ErrInvalidFilter                     = errors.New("meldbase: invalid filter")
	ErrInvalidUpdate                     = errors.New("meldbase: invalid update")
	ErrMutationLimit                     = errors.New("meldbase: mutation affected-row limit exceeded")
	ErrWriteConflict                     = errors.New("meldbase: write transaction snapshot conflicted")
	ErrWriteTransactionUnsupported       = errors.New("meldbase: write transactions require storage V2")
	ErrNotFound                          = errors.New("meldbase: document not found")
	ErrDuplicateID                       = errors.New("meldbase: duplicate document id")
	ErrInvalidCollection                 = errors.New("meldbase: invalid collection")
	ErrImmutableID                       = errors.New("meldbase: _id is immutable")
	ErrSlowConsumer                      = errors.New("meldbase: change consumer is too slow")
	ErrCorrupt                           = errors.New("meldbase: corrupt database")
	ErrUnsupportedFormat                 = errors.New("meldbase: unsupported storage format or required feature")
	ErrDuplicateKey                      = errors.New("meldbase: duplicate index key")
	ErrInvalidIndex                      = errors.New("meldbase: invalid index")
	ErrCompoundIndexUnsupported          = errors.New("meldbase: compound or descending indexes require storage V2 or memory")
	ErrDurability                        = errors.New("meldbase: durability failure; writes are disabled")
	ErrInvalidDelta                      = errors.New("meldbase: invalid query delta")
	ErrHistoryLost                       = errors.New("meldbase: requested history is no longer retained")
	ErrMigrationUnsupported              = errors.New("meldbase: migration requires an open V1 durable database")
	ErrMigrationDestinationExists        = errors.New("meldbase: migration destination already exists")
	ErrCompactionUnsupported             = errors.New("meldbase: compaction requires an open V2 database")
	ErrCompactionDestinationExists       = errors.New("meldbase: compaction destination already exists")
	ErrReclamationUnsupported            = errors.New("meldbase: page reclamation requires an open V2 database")
	ErrInvalidReclamationOptions         = errors.New("meldbase: invalid reclamation options")
	ErrReclamationConflict               = errors.New("meldbase: online reclamation conflicted with concurrent writes")
	ErrBackupUnsupported                 = errors.New("meldbase: physical backup requires an open V2 database")
	ErrBackupDestinationExists           = errors.New("meldbase: backup destination already exists or is the source")
	ErrVerificationUnsupported           = errors.New("meldbase: verification requires an existing V2 database")
	ErrDatabaseLocked                    = errors.New("meldbase: database is locked by another process")
	ErrRollbackDetected                  = errors.New("meldbase: database rollback detected")
	ErrDatabaseIdentity                  = errors.New("meldbase: unexpected database identity")
	ErrInsecureFileMode                  = errors.New("meldbase: database file permissions are not owner-private")
	ErrRollbackAnchorRequired            = errors.New("meldbase: rollback anchor is missing; explicit initialization is required")
	ErrRollbackAnchor                    = errors.New("meldbase: rollback anchor durability failure")
	ErrInvalidRollbackProtection         = errors.New("meldbase: invalid rollback protection options")
	ErrRecoveryRequired                  = errors.New("meldbase: startup recovery required by the selected policy")
	ErrInvalidResourceLimits             = errors.New("meldbase: invalid resource limits")
	ErrResourceLimit                     = errors.New("meldbase: resource limit exceeded")
	ErrIndexBuildUnsupported             = errors.New("meldbase: resumable index builds require storage V2")
	ErrIndexBuildNotFound                = errors.New("meldbase: index build not found")
	ErrIndexBuildExists                  = errors.New("meldbase: index build already exists")
	ErrIndexBuildFailed                  = errors.New("meldbase: index build is in a terminal failed state")
	ErrInvalidIndexBuildSchedulerOptions = errors.New("meldbase: invalid index build scheduler options")
	ErrIndexBuildSchedulerRunning        = errors.New("meldbase: index build scheduler already running")
	ErrInvalidCommitCoordinatorOptions   = errors.New("meldbase: invalid V2 commit coordinator options")
	ErrInvalidReplayDeliveryTimeout      = errors.New("meldbase: invalid V2 replay delivery timeout")
	ErrDurableConsumerUnsupported        = errors.New("meldbase: durable change consumers require storage V2")
	ErrDurableConsumerExists             = errors.New("meldbase: durable change consumer already exists")
	ErrDurableConsumerNotFound           = errors.New("meldbase: durable change consumer not found")
	ErrReplicaReadOnly                   = errors.New("meldbase: follower is read-only")
	ErrReplicaSequence                   = errors.New("meldbase: follower change token is not the next sequence")
	ErrReplicaProtocol                   = errors.New("meldbase: invalid replication protocol frame")
	ErrReplicaSourceActive               = errors.New("meldbase: replication source consumer already has an active session")
	ErrReplicaPromotionAuthority         = errors.New("meldbase: follower promotion requires an external fencing authority")
	ErrReplicaPromotionFence             = errors.New("meldbase: follower promotion fence does not match local state")
	ErrReplicaPromotionWriteFence        = errors.New("meldbase: follower promotion requires a configured primary write fence")
	ErrReplicaPromoted                   = errors.New("meldbase: follower has been promoted and no longer accepts replication")
	ErrPrimaryWriteFence                 = errors.New("meldbase: primary write fence rejected the commit")
	// ErrCommitOutcomeUnknown means cancellation or a lost caller connection
	// raced an already-admitted durable write. Callers must reconcile by the
	// returned document ID(s), rather than retrying business logic blindly.
	ErrCommitOutcomeUnknown = errors.New("meldbase: commit outcome is unknown; do not retry blindly")
)
View Source 
var DefaultQueryLimits = QueryLimits{
	MaxWireBytes: 1 << 20, MaxDepth: 16, MaxNodes: 128,
	MaxArrayItems: 256, MaxValueBytes: 16_384, MaxSortFields: 4,
	MaxLimit: 10_000,
}
View Source 
var (
	ErrDiagnosticsActive = errors.New("meldbase: diagnostics are already active")
)

Functions

func MarshalQuerySpecJSON 

func MarshalQuerySpecJSON(query QuerySpec) ([]byte, error)

MarshalQuerySpecJSON emits the canonical, data-only wire representation used for transport fingerprints and cross-language conformance.

func MarshalReplicationFrame 

func MarshalReplicationFrame(frame ReplicationFrame, limits ReplicationFrameLimits) ([]byte, error)

MarshalReplicationFrame returns a strict JSON frame with canonical document images encoded as base64 of the storage-independent typed document codec. It is suitable for WebSocket binary/text messages, QUIC streams or framed RPC, but it does not provide authentication or encryption itself.

func MarshalWireDocument 

func MarshalWireDocument(document Document) ([]byte, error)

func MarshalWireValue 

func MarshalWireValue(value Value) ([]byte, error)

func ValidateStrictJSON 

func ValidateStrictJSON(data []byte, maxBytes int) error

ValidateStrictJSON rejects oversized, trailing, deeply nested, and duplicate-key JSON before a transport decodes it into structs or maps.

Types

type ArchiveV2Bootstrap 

type ArchiveV2Bootstrap struct {
	Backup          BackupV2Result
	CheckpointToken uint64
	SnapshotToken   uint64
}

ArchiveV2Bootstrap binds an exact verified physical V2 snapshot to the durable database change feed that was pinned before that snapshot began.

A receiver must persist and verify Backup, then drain and Ack every batch up through SnapshotToken without applying it (the snapshot already contains those effects). It can then apply and Ack later batches in order. This avoids the bootstrap/tail gap without inventing a second, weaker history contract.

type BackupStats 

type BackupStats struct {
	Active       uint64        `json:"active"`
	Attempts     uint64        `json:"attempts"`
	Completed    uint64        `json:"completed"`
	Failed       uint64        `json:"failed"`
	LastBytes    uint64        `json:"lastBytes"`
	LastDuration time.Duration `json:"lastDurationNanos"`
}

type BackupV2Result 

type BackupV2Result struct {
	Bytes          uint64 `json:"bytes"`
	Pages          uint64 `json:"pages"`
	CommitSequence uint64 `json:"commitSequence"`
	MetaGeneration uint64 `json:"metaGeneration"`
	DatabaseIDHex  string `json:"databaseIdHex"`
	SHA256         string `json:"sha256"`
}

func ImportV2PhysicalBackup 

func ImportV2PhysicalBackup(ctx context.Context, source io.Reader, destination string, expected BackupV2Result, options PhysicalBackupImportOptions) (BackupV2Result, error)

ImportV2PhysicalBackup receives one exact BackupV2 artifact into a new local path. It writes a private temporary file, checks the claimed byte count and SHA-256 while streaming, runs the complete offline V2 graph/index verifier, then publishes with the same no-overwrite link-and-directory-sync commit point as backup and migration.

source is intentionally transport-neutral. A WebSocket, HTTP response, QUIC stream, or removable-media reader may supply it, but transport cancellation must close or honor ctx itself: a generic io.Reader cannot be interrupted while blocked in Read. The destination is never opened as a writable DB by this function; callers normally open the successfully imported file through OpenV2Follower before applying a replication tail.

type Change 

type Change struct {
	Collection string
	Operation  Operation
	DocumentID DocumentID
	Before     *Document
	After      *Document
	Index      *IndexDefinition

	// ChangedPaths is the sorted, deduplicated set of document paths changed by
	// an Update operation when the writer can prove it. A nil value means the
	// changed field set is unknown and consumers must conservatively treat the
	// whole document as affected. Insert, delete and catalog changes intentionally
	// use that conservative form: their membership and visible payload may change
	// through any query path.
	//
	// The slice is immutable once a Change enters the dispatcher. Public watcher
	// deliveries receive an independent copy via cloneChange.
	ChangedPaths []string
	// contains filtered or unexported fields
}

type ChangeBatch 

type ChangeBatch struct {
	Token   uint64
	Changes []Change
}

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

func (*Collection) CreateIndex 

func (c *Collection) CreateIndex(ctx context.Context, name string, fields []IndexField, options IndexOptions) error

func (*Collection) CreateIndexOnline 

func (c *Collection) CreateIndexOnline(ctx context.Context, name string, fields []IndexField, options IndexOptions) (IndexBuildID, error)

CreateIndexOnline starts and runs one durable build to publication. A context cancellation leaves the returned build discoverable through IndexBuilds; use StartIndexBuild when the ID must be known before execution.

func (*Collection) DeleteMany 

func (c *Collection) DeleteMany(ctx context.Context, filter Filter) (DeleteResult, error)

func (*Collection) DeleteManyQuery 

func (c *Collection) DeleteManyQuery(ctx context.Context, query QuerySpec) (DeleteResult, error)

func (*Collection) DeleteManyQueryLimited 

func (c *Collection) DeleteManyQueryLimited(ctx context.Context, query QuerySpec, maxAffected int) (DeleteResult, error)

DeleteManyQueryLimited atomically rejects the whole mutation when more than maxAffected documents match.

func (*Collection) DeleteOne 

func (c *Collection) DeleteOne(ctx context.Context, filter Filter) (DeleteResult, error)

func (*Collection) DeleteOneQuery 

func (c *Collection) DeleteOneQuery(ctx context.Context, query QuerySpec) (DeleteResult, error)

func (*Collection) Explain 

func (c *Collection) Explain(ctx context.Context, filter Filter) (ExplainResult, error)

func (*Collection) Find 

func (c *Collection) Find(ctx context.Context, filter Filter, options ...QueryOptions) (*Cursor, error)

func (*Collection) FindOne 

func (c *Collection) FindOne(ctx context.Context, filter Filter) (Document, error)

func (*Collection) FindQuery 

func (c *Collection) FindQuery(ctx context.Context, query QuerySpec) (*Cursor, error)

func (*Collection) InsertMany 

func (c *Collection) InsertMany(ctx context.Context, documents []Document) ([]DocumentID, error)

InsertMany validates IDs, documents, and all unique-index keys before writing one WAL record. Either the entire batch becomes visible or none of it does.

func (*Collection) InsertOne 

func (c *Collection) InsertOne(ctx context.Context, document Document) (DocumentID, error)

func (*Collection) SnapshotQuery 

func (c *Collection) SnapshotQuery(ctx context.Context, query QuerySpec) (QuerySnapshot, error)

func (*Collection) StartIndexBuild 

func (c *Collection) StartIndexBuild(ctx context.Context, name string, fields []IndexField, options IndexOptions) (IndexBuildID, error)

StartIndexBuild creates a durable private shadow index. It is Storage V2 only and does not scan documents or make the index query-visible.

func (*Collection) SubscribeQuery 

func (c *Collection) SubscribeQuery(ctx context.Context, query QuerySpec, buffer int) (*QuerySubscription, error)

func (*Collection) SubscribeQueryDeltas 

func (c *Collection) SubscribeQueryDeltas(ctx context.Context, query QuerySpec, buffer int) (*QueryDeltaSubscription, error)

func (*Collection) UpdateMany 

func (c *Collection) UpdateMany(ctx context.Context, filter Filter, update Update) (UpdateResult, error)

func (*Collection) UpdateManyQuery 

func (c *Collection) UpdateManyQuery(ctx context.Context, query QuerySpec, mutation MutationSpec) (UpdateResult, error)

func (*Collection) UpdateManyQueryLimited 

func (c *Collection) UpdateManyQueryLimited(ctx context.Context, query QuerySpec, mutation MutationSpec, maxAffected int) (UpdateResult, error)

UpdateManyQueryLimited atomically rejects the whole mutation when more than maxAffected documents match. A non-positive limit is invalid here so callers cannot accidentally disable a server-owned safety bound.

func (*Collection) UpdateOne 

func (c *Collection) UpdateOne(ctx context.Context, filter Filter, update Update) (UpdateResult, error)

func (*Collection) UpdateOneQuery 

func (c *Collection) UpdateOneQuery(ctx context.Context, query QuerySpec, mutation MutationSpec) (UpdateResult, error)

type CommitStats 

type CommitStats struct {
	Total   uint64 `json:"total"`
	Changes uint64 `json:"changes"`
}

type CompactionStats 

type CompactionStats struct {
	Active       uint64        `json:"active"`
	Attempts     uint64        `json:"attempts"`
	Completed    uint64        `json:"completed"`
	Failed       uint64        `json:"failed"`
	InputBytes   uint64        `json:"inputBytes"`
	OutputBytes  uint64        `json:"outputBytes"`
	LastDuration time.Duration `json:"lastDurationNanos"`
}

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

func (*Cursor) All 

func (c *Cursor) All(ctx context.Context) ([]Document, error)

func (*Cursor) Close 

func (c *Cursor) Close() error

Close releases a pinned storage snapshot held by a lazy cursor. It is safe to call repeatedly. Exhaustion, limit completion, errors and context cancellation close automatically; callers that stop early must close explicitly.

func (*Cursor) Next 

func (c *Cursor) Next(ctx context.Context) (Document, bool, error)

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func New 

func New() *DB

func NewWithOptions 

func NewWithOptions(options DatabaseOptions) (*DB, error)

NewWithOptions creates an in-memory database with explicit resource limits.

func Open 

func Open(path string) (*DB, error)

Open opens an existing V1 or V2 database after read-only format detection. A missing or zero-length path creates V2, the current default format. It never migrates or rewrites an existing V1 database implicitly.

func OpenV1 

func OpenV1(path string) (*DB, error)

OpenV1 explicitly creates or opens the legacy page-checkpoint plus WAL format. Existing applications normally use Open, which still recognizes V1 without migrating it. New databases should use Open or OpenV2.

func OpenV1WithOptions 

func OpenV1WithOptions(path string, options V1Options) (*DB, error)

OpenV1WithOptions explicitly opens the legacy page-checkpoint plus WAL format with a bounded automatic checkpoint policy.

func OpenV2 

func OpenV2(path string) (*DB, error)

OpenV2 explicitly creates or opens Storage V2. It never interprets or migrates a V1 file. Open performs read-only format selection when callers want to support both generations.

func OpenV2WithOptions 

func OpenV2WithOptions(path string, options V2Options) (*DB, error)

func OpenWithOptions 

func OpenWithOptions(path string, options OpenOptions) (*DB, error)

func (*DB) AbortIndexBuild 

func (db *DB) AbortIndexBuild(ctx context.Context, id IndexBuildID) error

func (*DB) AcquireReplicationSourceLease 

func (db *DB) AcquireReplicationSourceLease(name string) (*ReplicationSourceLease, error)

AcquireReplicationSourceLease reserves name until Release. Callers must authenticate and derive name outside untrusted replication frames. The lease is intentionally DB-local: a database file already has one writer process, while primary election remains an external control-plane responsibility.

func (*DB) BackupV2 

func (db *DB) BackupV2(ctx context.Context, destination string) (result BackupV2Result, resultErr error)

BackupV2 writes an exact, verified physical copy to a new path. It preserves database identity and Commit Log history, so the result is a restore artifact rather than an independent writable fork. The source writer is blocked for the copy duration; readers remain available.

func (*DB) BeginV2Archive 

func (db *DB) BeginV2Archive(ctx context.Context, name, destination string, buffer int) (ArchiveV2Bootstrap, *DurableDatabaseChangeSubscription, error)

BeginV2Archive creates a durable database-wide checkpoint and a verified physical snapshot. The checkpoint is established first, so Commit Log retention preserves every token needed to bridge from CheckpointToken to the returned SnapshotToken, even while writes continue before the snapshot's short read barrier begins.

This is deliberately a transport-neutral bootstrap primitive. It does not copy files to another machine, open a writable follower, or acknowledge a token on the caller's behalf. Those actions have different ownership and failure domains and must be explicit in the eventual wire protocol.

func (*DB) CanResumeFrom 

func (db *DB) CanResumeFrom(token uint64) bool

func (*DB) Close 

func (db *DB) Close() error

func (*DB) Collection 

func (db *DB) Collection(name string) *Collection

func (*DB) CommitCoordinatorStats 

func (db *DB) CommitCoordinatorStats() V2CommitCoordinatorStats

CommitCoordinatorStats returns a bounded snapshot of the optional V2 admission scheduler. It performs no I/O. DBStats includes the same snapshot in the versioned admin schema; this method is convenient for direct callers.

func (*DB) CompactToV2 

func (db *DB) CompactToV2(ctx context.Context, destination string) (resultErr error)

CompactToV2 writes one current logical V2 snapshot into a new, atomically published V2 file. It never overwrites destination or mutates the source. Writes which commit after the source snapshot is pinned may continue and are intentionally absent from the destination. The compacted database receives a new identity and commit-log history, so callers must treat every old resume token as invalid.

func (*DB) CompactToV2WithOptions 

func (db *DB) CompactToV2WithOptions(ctx context.Context, destination string, options V2DestinationOptions) (resultErr error)

CompactToV2WithOptions is CompactToV2 with an explicit destination quota.

func (*DB) CreateDurableCollectionChanges 

func (db *DB) CreateDurableCollectionChanges(ctx context.Context, name, collection string, afterToken uint64, buffer int) (*DurableChangeSubscription, error)

CreateDurableCollectionChanges creates a durable, collection-scoped document feed at afterToken. The requested position must still be retained. A logical name may be reused for a different collection without collision; the storage checkpoint identity is derived from both values.

func (*DB) CreateDurableDatabaseChanges 

func (db *DB) CreateDurableDatabaseChanges(ctx context.Context, name string, afterToken uint64, buffer int) (*DurableDatabaseChangeSubscription, error)

CreateDurableDatabaseChanges creates a named durable feed after afterToken. The requested position must still be retained. Names share no namespace with collection-scoped consumers, so one application can use the same logical name for both an outbox and a database archive without a checkpoint clash.

func (*DB) DatabaseID 

func (db *DB) DatabaseID() [16]byte

DatabaseID returns the stable, non-secret V2/V1 database namespace used to bind resume and replication protocols. It is not an authentication token.

func (*DB) DatabaseIdentity 

func (db *DB) DatabaseIdentity() [16]byte

func (*DB) DeleteDurableCollectionChanges 

func (db *DB) DeleteDurableCollectionChanges(ctx context.Context, name, collection string) error

DeleteDurableCollectionChanges explicitly removes one retained checkpoint. Callers should stop every owner of that logical consumer first; an already open stream retains only its temporary process-local pin until it closes.

func (*DB) DeleteDurableDatabaseChanges 

func (db *DB) DeleteDurableDatabaseChanges(ctx context.Context, name string) error

DeleteDurableDatabaseChanges removes a named database-wide checkpoint. Stop all active owners first; deletion deliberately makes future resume fail.

func (*DB) DiagnosticSnapshotAfter 

func (db *DB) DiagnosticSnapshotAfter(after uint64, limit int) DiagnosticSnapshot

DiagnosticSnapshotAfter reads the currently active diagnostic session. It allows long-lived admin handlers to follow a safely replaced session.

func (*DB) EnableDiagnostics 

func (db *DB) EnableDiagnostics(options DiagnosticsOptions) (*Diagnostics, error)

func (*DB) IndexBuild 

func (db *DB) IndexBuild(id IndexBuildID) (IndexBuildStatus, error)

func (*DB) IndexBuilds 

func (db *DB) IndexBuilds() ([]IndexBuildStatus, error)

IndexBuilds returns all durable unfinished builds. The result survives a clean close, process crash, and reopen.

func (*DB) MeldbaseSystemRecordBackend 

func (db *DB) MeldbaseSystemRecordBackend() systemrecord.Backend

MeldbaseSystemRecordBackend is internal plumbing for first-party packages such as server. Its return type lives under internal/, so it is deliberately unavailable as an application-facing generic key/value API.

func (*DB) MeldbaseSystemWrite 

func (db *DB) MeldbaseSystemWrite(ctx context.Context, systemMutation systemrecord.Mutation, build func(*WriteTransaction) ([]byte, error)) (systemrecord.Result, bool, error)

MeldbaseSystemWrite runs build against one immutable V2 snapshot without holding the database writer lock. If build succeeds and its point read set is still valid, its business changes and systemMutation commit in one V2 generation. The bool reports whether a composite commit was attempted; false means build produced no business change or lost optimistic validation.

This method is first-party plumbing: the systemrecord parameter is internal, preventing external applications from using the private keyspace.

func (*DB) MigrateToV2 

func (db *DB) MigrateToV2(ctx context.Context, destination string) error

MigrateToV2 writes a consistent logical snapshot of an open V1 durable DB to a new V2 file. The source remains open and unchanged. The destination must not exist and is published only after V2 reopen and semantic verification. A successful migration deliberately has a new database identity, invalidating every V1 resume token rather than mapping it onto unrelated V2 commit history.

func (*DB) MigrateToV2WithOptions 

func (db *DB) MigrateToV2WithOptions(ctx context.Context, destination string, options V2DestinationOptions) error

MigrateToV2WithOptions is MigrateToV2 with an explicit destination quota.

func (*DB) OpenDurableCollectionChanges 

func (db *DB) OpenDurableCollectionChanges(ctx context.Context, name, collection string, buffer int) (*DurableChangeSubscription, error)

OpenDurableCollectionChanges reopens an existing durable collection feed at its stored checkpoint. If retention cannot satisfy that position it returns ErrHistoryLost; it never silently starts from a newer token.

func (*DB) OpenDurableDatabaseChanges 

func (db *DB) OpenDurableDatabaseChanges(ctx context.Context, name string, buffer int) (*DurableDatabaseChangeSubscription, error)

OpenDurableDatabaseChanges resumes a named durable feed from its persisted checkpoint. ErrHistoryLost is returned instead of silently starting at a later token when the required Commit Log window is gone.

func (*DB) OpenQueryReplay 

func (db *DB) OpenQueryReplay(ctx context.Context, collection string, query QuerySpec, afterToken uint64, buffer int) (*QueryReplaySubscription, error)

func (*DB) OperationalState 

func (db *DB) OperationalState() OperationalState

func (*DB) ReclaimV2Pages 

func (db *DB) ReclaimV2Pages(ctx context.Context) (result ReclaimV2Result, resultErr error)

ReclaimV2Pages audits both valid Meta roots and every active snapshot/replay lease, then makes only unreachable pages available to future COW commits. The free pool is process-local and safely reconstructed by another call on reopen.

func (*DB) ReclaimV2PagesWithOptions 

func (db *DB) ReclaimV2PagesWithOptions(ctx context.Context, options ReclaimV2Options) (result ReclaimV2Result, resultErr error)

ReclaimV2PagesWithOptions runs explicit synchronous or low-pause optimistic reclamation. Online mode is opt-in and may return an error wrapping ErrReclamationConflict when every bounded attempt overlaps a commit.

func (*DB) RecoveryReport 

func (db *DB) RecoveryReport() RecoveryReport

RecoveryReport returns the receipt captured by the successful constructor. It performs no I/O and never changes after the DB is opened.

func (*DB) ResourceLimits 

func (db *DB) ResourceLimits() ResourceLimits

ResourceLimits returns the immutable normalized limits selected at open.

func (*DB) ResumeIndexBuild 

func (db *DB) ResumeIndexBuild(ctx context.Context, id IndexBuildID) error

ResumeIndexBuild scans bounded batches, catches up retained commits, and atomically publishes the index. Only one caller should resume a given build; stale concurrent callers receive ErrWriteConflict from durable CAS checks.

func (*DB) RunWriteTransaction 

func (db *DB) RunWriteTransaction(ctx context.Context, build func(*WriteTransaction) error) error

RunWriteTransaction executes build against one immutable Storage V2 snapshot and atomically publishes all staged point mutations if every document read by the callback still matches. The callback runs without the database writer lock. A conflicting point write returns ErrWriteConflict; callbacks are never retried because they may contain application side effects.

A successful callback with no effective changes is a successful no-op and does not advance the commit sequence. The transaction is invalid as soon as the callback returns. Normal DB and Collection methods must not be called from inside build.

func (*DB) StartIndexBuildScheduler 

func (db *DB) StartIndexBuildScheduler(parent context.Context, options IndexBuildSchedulerOptions) (*IndexBuildScheduler, error)

func (*DB) StartV2Maintenance 

func (db *DB) StartV2Maintenance(parent context.Context, options V2MaintenanceOptions) (*V2Maintenance, error)

func (*DB) Stats 

func (db *DB) Stats() DBStats

func (*DB) Sync 

func (db *DB) Sync() error

func (*DB) WatchChanges 

func (db *DB) WatchChanges(ctx context.Context, collection string, buffer int) (<-chan ChangeBatch, <-chan error, error)

type DBStats 

type DBStats struct {
	CapturedAt           time.Time     `json:"capturedAt"`
	StartedAt            time.Time     `json:"startedAt"`
	Uptime               time.Duration `json:"uptimeNanos"`
	Closed               bool          `json:"closed"`
	WritesDisabled       bool          `json:"writesDisabled"`
	Durable              bool          `json:"durable"`
	CommitSequence       uint64        `json:"commitSequence"`
	Collections          uint64        `json:"collections"`
	Documents            uint64        `json:"documents"`
	Indexes              uint64        `json:"indexes"`
	ActiveChangeWatchers uint64        `json:"activeChangeWatchers"`

	Commits           CommitStats              `json:"commits"`
	Transactions      WriteTransactionStats    `json:"writeTransactions"`
	Queries           QueryStats               `json:"queries"`
	Realtime          RealtimeStats            `json:"realtime"`
	CommitCoordinator V2CommitCoordinatorStats `json:"commitCoordinator"`
	PrimaryWriteFence V2PrimaryWriteFenceStats `json:"primaryWriteFence"`
	Durability        DurabilityStats          `json:"durability"`
	Storage           StorageStats             `json:"storage"`
	Compaction        CompactionStats          `json:"compaction"`
	Reclamation       ReclamationStats         `json:"reclamation"`
	Backup            BackupStats              `json:"backup"`
	Diagnostics       DiagnosticStats          `json:"diagnostics"`
	Recovery          RecoveryReport           `json:"recovery"`
	Resources         ResourceStats            `json:"resources"`
	IndexBuilds       IndexBuildStats          `json:"indexBuilds"`
}

DBStats is a point-in-time, allocation-bounded view of database health. Counters are process-lifetime values and reset when the database is reopened. Persistent state such as CommitSequence is read from the database itself.

Stats deliberately exposes no user values, document IDs, query parameters, or callbacks. It is safe for an admin sampler to call periodically, but it is not intended to be called on every database operation.

type DatabaseOptions 

type DatabaseOptions struct{ ResourceLimits ResourceLimits }

DatabaseOptions configures an in-memory database.

type DeleteResult 

type DeleteResult struct{ DeletedCount int64 }

type DiagnosticEvent 

type DiagnosticEvent struct {
	Sequence          uint64            `json:"sequence"`
	CapturedAt        time.Time         `json:"capturedAt"`
	Kind              DiagnosticKind    `json:"kind"`
	Outcome           DiagnosticOutcome `json:"outcome"`
	ErrorClass        string            `json:"errorClass,omitempty"`
	Stage             string            `json:"stage,omitempty"`
	Duration          time.Duration     `json:"durationNanos"`
	DocumentsExamined uint64            `json:"documentsExamined,omitempty"`
	DocumentsReturned uint64            `json:"documentsReturned,omitempty"`
	Changes           uint64            `json:"changes,omitempty"`
	Slow              bool              `json:"slow"`
	Sampled           bool              `json:"sampled"`
}

type DiagnosticKind 

type DiagnosticKind string
const (
	DiagnosticQuery  DiagnosticKind = "query"
	DiagnosticCommit DiagnosticKind = "commit"
)

type DiagnosticOutcome 

type DiagnosticOutcome string
const (
	DiagnosticSuccess  DiagnosticOutcome = "success"
	DiagnosticFailure  DiagnosticOutcome = "failure"
	DiagnosticCanceled DiagnosticOutcome = "canceled"
)

type DiagnosticSnapshot 

type DiagnosticSnapshot struct {
	Version    uint32            `json:"version"`
	Session    uint64            `json:"session"`
	StartedAt  time.Time         `json:"startedAt"`
	CapturedAt time.Time         `json:"capturedAt"`
	Stats      DiagnosticStats   `json:"stats"`
	Events     []DiagnosticEvent `json:"events"`
	Truncated  bool              `json:"truncated"`
	HasMore    bool              `json:"hasMore"`
}

type DiagnosticStats 

type DiagnosticStats struct {
	Enabled         bool   `json:"enabled"`
	Capacity        uint64 `json:"capacity"`
	Retained        uint64 `json:"retained"`
	Recorded        uint64 `json:"recorded"`
	Overwritten     uint64 `json:"overwritten"`
	QueriesObserved uint64 `json:"queriesObserved"`
	CommitsObserved uint64 `json:"commitsObserved"`
}

type Diagnostics 

type Diagnostics struct {
	// contains filtered or unexported fields
}

Diagnostics owns a fixed-capacity event ring. Close disables future timing and recording but keeps the retained snapshot readable by its owner.

func (*Diagnostics) Close 

func (d *Diagnostics) Close() error

func (*Diagnostics) DiagnosticSnapshotAfter 

func (d *Diagnostics) DiagnosticSnapshotAfter(after uint64, limit int) DiagnosticSnapshot

DiagnosticSnapshotAfter lets a fixed Diagnostics handle also satisfy admin diagnostic-source contracts.

func (*Diagnostics) Snapshot 

func (d *Diagnostics) Snapshot() DiagnosticSnapshot

func (*Diagnostics) SnapshotAfter 

func (d *Diagnostics) SnapshotAfter(after uint64, limit int) DiagnosticSnapshot

SnapshotAfter returns retained events with sequence greater than after in chronological order. A non-positive limit means the ring capacity. Truncated reports that after predates the oldest retained sequence; HasMore asks the caller to continue from the last returned sequence.

func (*Diagnostics) Stats 

func (d *Diagnostics) Stats() DiagnosticStats

type DiagnosticsOptions 

type DiagnosticsOptions struct {
	Capacity            int
	SlowQueryThreshold  time.Duration
	SlowCommitThreshold time.Duration
	SampleEvery         uint64
	RecordAll           bool
	ExcludeFailures     bool
}

DiagnosticsOptions controls opt-in detailed events. Defaults retain 256 events and record failed, >=50ms queries and >=100ms durable commits. Setting RecordAll is intended only for short development sessions. SampleEvery adds a deterministic one-in-N sample of otherwise fast successful operations.

type Document 

type Document map[string]Value

func NewDocument 

func NewDocument(fields map[string]any) (Document, error)

func UnmarshalWireDocument 

func UnmarshalWireDocument(data []byte, limits QueryLimits) (Document, error)

func UnmarshalWireInputDocument 

func UnmarshalWireInputDocument(data []byte, limits QueryLimits) (Document, error)

func (Document) Clone 

func (d Document) Clone() Document

func (Document) Equal 

func (d Document) Equal(other Document) bool

func (Document) ID 

func (d Document) ID() (DocumentID, bool)

func (Document) Validate 

func (d Document) Validate() error

Validate checks every nested field and value before a document crosses a storage or transport boundary.

type DocumentCacheStats 

type DocumentCacheStats struct {
	CapacityEntries uint64 `json:"capacityEntries"`
	CapacityBytes   uint64 `json:"capacityBytes"`
	Entries         uint64 `json:"entries"`
	Bytes           uint64 `json:"bytes"`
	Hits            uint64 `json:"hits"`
	Misses          uint64 `json:"misses"`
	Evictions       uint64 `json:"evictions"`
}

type DocumentID 

type DocumentID [16]byte

func NewDocumentID 

func NewDocumentID() (DocumentID, error)

func ParseDocumentID 

func ParseDocumentID(s string) (DocumentID, error)

func (DocumentID) IsZero 

func (id DocumentID) IsZero() bool

func (DocumentID) String 

func (id DocumentID) String() string

type DurabilityStats 

type DurabilityStats struct {
	WALAppends           uint64        `json:"walAppends"`
	WALPayloadBytes      uint64        `json:"walPayloadBytes"`
	WALCurrentBytes      uint64        `json:"walCurrentBytes"`
	WALCurrentCommits    uint64        `json:"walCurrentCommits"`
	WALAppendFailures    uint64        `json:"walAppendFailures"`
	WALAppendNanos       uint64        `json:"walAppendNanos"`
	WALAppendMaxLatency  time.Duration `json:"walAppendMaxLatencyNanos"`
	CheckpointAttempts   uint64        `json:"checkpointAttempts"`
	CheckpointsCompleted uint64        `json:"checkpointsCompleted"`
	CheckpointFailures   uint64        `json:"checkpointFailures"`
	AutomaticCheckpoints uint64        `json:"automaticCheckpoints"`
	CheckpointNanos      uint64        `json:"checkpointNanos"`
	CheckpointMaxLatency time.Duration `json:"checkpointMaxLatencyNanos"`
}

type DurableChangeBatch 

type DurableChangeBatch struct {
	Token   uint64
	Changes []Change
}

DurableChangeBatch is one globally ordered Commit Log position projected to one collection. Changes is empty when another collection or private catalog change advanced the durable position; callers must still Ack that Token after processing it so the checkpoint can advance without pinning history forever.

This is deliberately a document-change feed, not a full replication protocol: it does not expose private System records, index definitions, collection lifecycle or raw storage bytes.

type DurableChangeSubscription 

type DurableChangeSubscription struct {
	Batches <-chan DurableChangeBatch
	Errors  <-chan error
	// contains filtered or unexported fields
}

DurableChangeSubscription is a pull/acknowledge bridge over a V2 durable checkpoint. Batches remain ordered. Ack must be called only after the consumer's external side effect for that token is durable.

func (*DurableChangeSubscription) Ack 

func (subscription *DurableChangeSubscription) Ack(token uint64) error

func (*DurableChangeSubscription) Close 

func (subscription *DurableChangeSubscription) Close()

type DurableDatabaseChangeBatch 

type DurableDatabaseChangeBatch struct {
	Token         uint64
	TransactionID [16]byte
	CommittedAt   time.Time
	Changes       []Change
}

DurableDatabaseChangeBatch is one globally ordered V2 Commit Log position projected into public document and catalog events. It is the semantic source for archive and single-writer-follower protocols: callers must Ack only after the externally applied effect for Token is durable.

Private System records, raw pages, index-build progress and retention control records are deliberately excluded. A batch can therefore be empty when a private record advanced a retained position; it must still be Acked.

type DurableDatabaseChangeSubscription 

type DurableDatabaseChangeSubscription struct {
	Batches <-chan DurableDatabaseChangeBatch
	Errors  <-chan error
	// contains filtered or unexported fields
}

DurableDatabaseChangeSubscription is a crash-resumable pull/acknowledge feed over the complete public V2 database. It exposes collection creation, index publication and document changes in exact Commit Log order. It does not itself copy a bootstrap snapshot or apply changes to a follower; those transport and ownership contracts are intentionally separate.

func (*DurableDatabaseChangeSubscription) Ack 

func (subscription *DurableDatabaseChangeSubscription) Ack(token uint64) error

func (*DurableDatabaseChangeSubscription) Checkpoint 

func (subscription *DurableDatabaseChangeSubscription) Checkpoint() (uint64, error)

Checkpoint returns the consumer's last durable acknowledgement. Delivered batches do not move it; callers may use it to bind a replication hello to the exact retained position, rather than to a process-local queue position.

func (*DurableDatabaseChangeSubscription) Close 

func (subscription *DurableDatabaseChangeSubscription) Close()

type ExplainResult 

type ExplainResult struct {
	Stage, IndexName                string
	DocumentsExamined, KeysExamined int64
}

type Filter 

type Filter map[string]any

type FollowerPromotionAuthority 

type FollowerPromotionAuthority interface {
	AuthorizeFollowerPromotion(context.Context, FollowerPromotionRequest) (FollowerPromotionFence, error)
}

FollowerPromotionAuthority must make its returned fence durable before it returns. Implementations normally revoke a primary lease through a quorum controller or external consensus store.

type FollowerPromotionFence 

type FollowerPromotionFence struct {
	DatabaseID     [16]byte
	CommitSequence uint64
	Epoch          string
}

FollowerPromotionFence is a controller-issued, non-empty epoch proving the old primary's write authority was fenced for this database/token. Epoch is deliberately opaque to Meldbase: a controller may use it as an epoch ID or a compact signed lease certificate (as integrations/primarylease does). Meldbase does not invent a local substitute for that distributed safety decision.

type FollowerPromotionFenceBinder 

type FollowerPromotionFenceBinder interface {
	BindV2FollowerPromotion(context.Context, FollowerPromotionFence) error
}

FollowerPromotionFenceBinder binds one controller-issued promotion fence to the local V2 primary-write guard before a follower becomes writable. The binder may update caller-owned local lease/epoch state, but must not enable writes until it has accepted the exact fence. It runs on the promotion control path, outside the DB writer lock; unlike ValidateV2PrimaryWrite it may coordinate with the controller if the implementation needs to.

A promoted follower requires this interface in addition to V2PrimaryWriteFence. Otherwise an unrelated always-allow guard could make a one-time promotion certificate appear to grant permanent write authority.

type FollowerPromotionRequest 

type FollowerPromotionRequest struct {
	DatabaseID     [16]byte
	CommitSequence uint64
}

FollowerPromotionRequest is the exact local state an external fencing system must certify before this process can become writable primary.

type IndexBuildFailure 

type IndexBuildFailure string

type IndexBuildID 

type IndexBuildID [16]byte

IndexBuildID identifies one durable, resumable Storage V2 index build.

func ParseIndexBuildID 

func ParseIndexBuildID(value string) (IndexBuildID, error)

func (IndexBuildID) IsZero 

func (id IndexBuildID) IsZero() bool

func (IndexBuildID) MarshalText 

func (id IndexBuildID) MarshalText() ([]byte, error)

func (IndexBuildID) String 

func (id IndexBuildID) String() string

func (*IndexBuildID) UnmarshalText 

func (id *IndexBuildID) UnmarshalText(value []byte) error

type IndexBuildPhase 

type IndexBuildPhase string

type IndexBuildScheduler 

type IndexBuildScheduler struct {
	// contains filtered or unexported fields
}

func (*IndexBuildScheduler) Done 

func (scheduler *IndexBuildScheduler) Done() <-chan struct{}

func (*IndexBuildScheduler) Stats 

func (scheduler *IndexBuildScheduler) Stats() IndexBuildSchedulerStats

func (*IndexBuildScheduler) Stop 

func (scheduler *IndexBuildScheduler) Stop()

type IndexBuildSchedulerOptions 

type IndexBuildSchedulerOptions struct {
	PollInterval   time.Duration
	RunTimeout     time.Duration
	MaxConcurrency int
	RunImmediately bool
}

IndexBuildSchedulerOptions configures an explicit default-off runner. Each task receives a bounded time quantum, then yields durable progress so CRUD and other builds can proceed between quanta.

type IndexBuildSchedulerStats 

type IndexBuildSchedulerStats struct {
	Polls        uint64        `json:"polls"`
	Runs         uint64        `json:"runs"`
	Completed    uint64        `json:"completed"`
	Yielded      uint64        `json:"yielded"`
	MarkedFailed uint64        `json:"markedFailed"`
	Conflicts    uint64        `json:"conflicts"`
	Failed       uint64        `json:"failed"`
	Active       uint64        `json:"active"`
	LastDuration time.Duration `json:"lastDurationNanos"`
	LastError    string        `json:"lastError,omitempty"`
}

type IndexBuildStats 

type IndexBuildStats struct {
	Active               uint64        `json:"active"`
	Persistent           uint64        `json:"persistent"`
	Scanning             uint64        `json:"scanning"`
	CatchingUp           uint64        `json:"catchingUp"`
	Ready                uint64        `json:"ready"`
	PersistentFailed     uint64        `json:"persistentFailed"`
	RetentionLeaseActive bool          `json:"retentionLeaseActive"`
	RetentionPressure    bool          `json:"retentionPressure"`
	PersistentEntries    uint64        `json:"persistentEntries"`
	PersistentBytes      uint64        `json:"persistentBytes"`
	SchedulerRuns        uint64        `json:"schedulerRuns"`
	SchedulerYields      uint64        `json:"schedulerYields"`
	SchedulerFailures    uint64        `json:"schedulerFailures"`
	Attempts             uint64        `json:"attempts"`
	Completed            uint64        `json:"completed"`
	Failed               uint64        `json:"failed"`
	Retries              uint64        `json:"retries"`
	Conflicts            uint64        `json:"conflicts"`
	LastEntries          uint64        `json:"lastEntries"`
	LastBytes            uint64        `json:"lastBytes"`
	LastDuration         time.Duration `json:"lastDurationNanos"`
	MaxDuration          time.Duration `json:"maxDurationNanos"`
	// contains filtered or unexported fields
}

type IndexBuildStatus 

type IndexBuildStatus struct {
	ID              IndexBuildID      `json:"id"`
	Collection      string            `json:"collection"`
	Name            string            `json:"name"`
	Field           string            `json:"field"`
	Fields          []IndexField      `json:"fields"`
	Unique          bool              `json:"unique"`
	Phase           IndexBuildPhase   `json:"phase"`
	Failure         IndexBuildFailure `json:"failure,omitempty"`
	SourceSequence  uint64            `json:"sourceSequence"`
	AppliedSequence uint64            `json:"appliedSequence"`
	EntryCount      uint64            `json:"entryCount"`
	CanonicalBytes  uint64            `json:"canonicalBytes"`
	CreatedAt       time.Time         `json:"createdAt"`
	UpdatedAt       time.Time         `json:"updatedAt"`
}

IndexBuildStatus is durable progress. EntryCount and CanonicalBytes describe the current private Secondary tree, not transient Go heap usage.

type IndexDefinition 

type IndexDefinition struct {
	Name, Field string
	Order       int
	Unique      bool
	// Fields is the ordered definition for compound/descending indexes. Field
	// and Order remain the compatibility mirror of Fields[0] for V1 records and
	// existing callers; new code must use indexDefinitionFields.
	Fields []IndexField
}

type IndexField 

type IndexField struct {
	Field string
	Order int
}

IndexField is one ordered component of an index definition. Order must be 1 (ascending) or -1 (descending); fields are evaluated left to right.

type IndexOptions 

type IndexOptions struct{ Unique bool }

IndexOptions controls complete-tuple uniqueness.

type Kind 

type Kind uint8
const (
	NullKind Kind = iota
	BoolKind
	Int64Kind
	Float64Kind
	StringKind
	BinaryKind
	TimeKind
	ArrayKind
	ObjectKind
	IDKind
)

type MutationSpec 

type MutationSpec struct {
	// contains filtered or unexported fields
}

func CompileUpdate 

func CompileUpdate(update Update) (MutationSpec, error)

func DecodeMutationSpecJSON 

func DecodeMutationSpecJSON(data []byte, limits QueryLimits) (MutationSpec, error)

func (MutationSpec) Apply 

func (m MutationSpec) Apply(document Document) (Document, error)

func (MutationSpec) Paths 

func (m MutationSpec) Paths() []string

type OpenOptions 

type OpenOptions struct {
	Recovery                RecoveryMode
	V1Checkpoint            V1CheckpointPolicy
	V2CommitRetention       V2CommitRetentionPolicy
	V2ReplayDeliveryTimeout time.Duration
	V2CommitCoordinator     V2CommitCoordinatorOptions
	ResourceLimits          ResourceLimits
	V2StorageLimits         V2StorageLimits
	V2RollbackProtection    V2RollbackProtection
	// V2RequireGraphAudit performs a structural full-graph audit before a V2
	// open succeeds. It is ignored for V1, whose recovery has a distinct WAL
	// validation contract.
	V2RequireGraphAudit bool
	// V2RequirePrivateFileMode rejects an existing V2 database that grants
	// group or other Unix permission bits. It is ignored for V1.
	V2RequirePrivateFileMode bool
	// V2PrimaryWriteFence is forwarded only when Open selects V2. Open rejects
	// an existing legacy V1 file when this boundary is requested, so a caller
	// cannot silently lose primary-fence enforcement through format detection.
	V2PrimaryWriteFence V2PrimaryWriteFence
}

OpenOptions configures format-neutral Open. V1Checkpoint is ignored for V2; V2 retention/replay/storage fields are ignored for V1, while V2RollbackProtection is rejected for V1 so a requested safety boundary is never silently absent.

type Operation 

type Operation string
const (
	InsertOperation Operation = "insert"
	UpdateOperation Operation = "update"
	DeleteOperation Operation = "delete"
	// CreateCollectionOperation is emitted only by the durable database change
	// feed. Ordinary collection creation remains implicit for CRUD callers.
	CreateCollectionOperation Operation = "create_collection"
	CreateIndexOperation      Operation = "create_index"
)

type OperationalState 

type OperationalState struct {
	Readable bool `json:"readable"`
	Writable bool `json:"writable"`
}

OperationalState is a minimal, allocation-free serving-state snapshot. A fail-stop durability error preserves reads from the last committed state but disables writes; a closed database is neither readable nor writable.

type PageCacheStats 

type PageCacheStats struct {
	CapacityPages uint64 `json:"capacityPages"`
	ResidentPages uint64 `json:"residentPages"`
	Hits          uint64 `json:"hits"`
	Misses        uint64 `json:"misses"`
	Evictions     uint64 `json:"evictions"`
}

type PhysicalBackupImportOptions 

type PhysicalBackupImportOptions struct {
	MaxBytes uint64
}

PhysicalBackupImportOptions bounds an untrusted physical-backup stream before it can consume local disk. Zero selects the normal V2 file limit. Deployments with a deliberately larger V2 database must set MaxBytes explicitly on the receiving side; a sender never chooses that authority.

type PrimaryWriteFenceRequest 

type PrimaryWriteFenceRequest struct {
	DatabaseID         [16]byte
	NextCommitSequence uint64
}

PrimaryWriteFenceRequest binds a proposed primary mutation to this database identity and exact next logical commit sequence. A lease implementation must reject when its external authority/epoch/expiry no longer permits that write.

type QueryDelta 

type QueryDelta struct {
	FromToken  uint64
	Token      uint64
	Operations []QueryDeltaOperation
}

QueryDelta transforms exactly FromToken into Token. Operations are ordered: removals first, followed by reverse-order add/move anchors and document changes. Applying them in slice order is deterministic.

type QueryDeltaOperation 

type QueryDeltaOperation struct {
	Kind       QueryDeltaOperationKind
	DocumentID DocumentID
	BeforeID   DocumentID
	Document   Document
}

QueryDeltaOperation mutates an ordered query result. A zero BeforeID means the end of the result; database document IDs are never zero.

type QueryDeltaOperationKind 

type QueryDeltaOperationKind string
const (
	QueryDeltaRemove QueryDeltaOperationKind = "remove"
	QueryDeltaAdd    QueryDeltaOperationKind = "add_before"
	QueryDeltaMove   QueryDeltaOperationKind = "move_before"
	QueryDeltaChange QueryDeltaOperationKind = "change"
)

type QueryDeltaSubscription 

type QueryDeltaSubscription struct {
	Initial QuerySnapshot
	Deltas  <-chan QueryDelta
	Errors  <-chan error
	// contains filtered or unexported fields
}

QueryDeltaSubscription returns one safe initial snapshot and then ordered deltas. It is the preferred core stream for transports and reactive clients; QuerySubscription remains the full-snapshot compatibility adapter.

func (*QueryDeltaSubscription) Close 

func (s *QueryDeltaSubscription) Close()

type QueryLimits 

type QueryLimits struct {
	MaxWireBytes  int
	MaxDepth      int
	MaxNodes      int
	MaxArrayItems int
	MaxValueBytes int
	MaxSortFields int
	MaxLimit      int
}

type QueryOptions 

type QueryOptions struct {
	Sort  []SortField
	Skip  int
	Limit *int
}

type QueryReplaySource 

type QueryReplaySource interface {
	OpenQueryReplay(ctx context.Context, collection string, query QuerySpec, afterToken uint64, buffer int) (*QueryReplaySubscription, error)
}

QueryReplaySource atomically reconstructs a query at afterToken and tails later ordered revisions. Initial.Token must equal afterToken. Implementations return ErrHistoryLost when retention can no longer satisfy that contract.

type QueryReplaySubscription 

type QueryReplaySubscription struct {
	Initial QuerySnapshot
	Deltas  <-chan QueryDelta
	Errors  <-chan error
	// contains filtered or unexported fields
}

func (*QueryReplaySubscription) Close 

func (subscription *QueryReplaySubscription) Close()

type QuerySnapshot 

type QuerySnapshot struct {
	Token     uint64
	Documents []Document
}

func ApplyQueryDelta 

func ApplyQueryDelta(snapshot QuerySnapshot, delta QueryDelta) (QuerySnapshot, error)

ApplyQueryDelta strictly validates and applies an ordered delta without mutating the input snapshot.

type QuerySpec 

type QuerySpec struct {
	// contains filtered or unexported fields
}

func CompileQuery 

func CompileQuery(filter Filter, options QueryOptions) (QuerySpec, error)

func DecodeQuerySpecJSON 

func DecodeQuerySpecJSON(data []byte, limits QueryLimits) (QuerySpec, error)

func (QuerySpec) Capped 

func (q QuerySpec) Capped(max int) QuerySpec

func (QuerySpec) Constrain 

func (q QuerySpec) Constrain(policy QuerySpec) QuerySpec

Constrain applies a server-owned row predicate before the caller's sort and pagination. This is the safe composition point for authorization policies.

func (QuerySpec) Execute 

func (q QuerySpec) Execute(documents []Document) []Document

func (QuerySpec) HasModifiers 

func (q QuerySpec) HasModifiers() bool

func (QuerySpec) Limit 

func (q QuerySpec) Limit() (int, bool)

func (QuerySpec) Match 

func (q QuerySpec) Match(document Document) bool

func (QuerySpec) Paths 

func (q QuerySpec) Paths() []string

func (QuerySpec) Skip 

func (q QuerySpec) Skip() int

func (QuerySpec) Sort 

func (q QuerySpec) Sort() []SortField

type QueryStats 

type QueryStats struct {
	ActiveCursors     uint64 `json:"activeCursors"`
	Total             uint64 `json:"total"`
	Failed            uint64 `json:"failed"`
	CollectionScans   uint64 `json:"collectionScans"`
	IndexScans        uint64 `json:"indexScans"`
	IDLookups         uint64 `json:"idLookups"`
	DocumentsExamined uint64 `json:"documentsExamined"`
	DocumentsReturned uint64 `json:"documentsReturned"`
}

type QuerySubscription 

type QuerySubscription struct {
	Snapshots <-chan QuerySnapshot
	Errors    <-chan error
	// contains filtered or unexported fields
}

func (*QuerySubscription) Close 

func (s *QuerySubscription) Close()

type RealtimeStats 

type RealtimeStats struct {
	SharedViews            uint64 `json:"sharedViews"`
	QuerySubscribers       uint64 `json:"querySubscribers"`
	SharedViewReuses       uint64 `json:"sharedViewReuses"`
	IncrementalBatches     uint64 `json:"incrementalBatches"`
	IncrementalViewUpdates uint64 `json:"incrementalViewUpdates"`
	FullViewRecomputes     uint64 `json:"fullViewRecomputes"`
	QueueOverflows         uint64 `json:"queueOverflows"`
	PendingBatches         uint64 `json:"pendingBatches"`
	PendingChanges         uint64 `json:"pendingChanges"`
	PendingBytes           uint64 `json:"pendingBytes"`
	PendingBatchCapacity   uint64 `json:"pendingBatchCapacity"`
	PendingChangeCapacity  uint64 `json:"pendingChangeCapacity"`
	PendingByteCapacity    uint64 `json:"pendingByteCapacity"`
	WatcherPendingBytes    uint64 `json:"watcherPendingBytes"`
	WatcherByteCapacity    uint64 `json:"watcherByteCapacity"`
	DispatchPendingBatches uint64 `json:"dispatchPendingBatches"`
	DispatchPendingChanges uint64 `json:"dispatchPendingChanges"`
	DispatchPendingBytes   uint64 `json:"dispatchPendingBytes"`
	DispatchBatchCapacity  uint64 `json:"dispatchBatchCapacity"`
	DispatchChangeCapacity uint64 `json:"dispatchChangeCapacity"`
	DispatchByteCapacity   uint64 `json:"dispatchByteCapacity"`
	SharedDeltas           uint64 `json:"sharedDeltas"`
	DeltaDeliveries        uint64 `json:"deltaDeliveries"`
	DeltaOperations        uint64 `json:"deltaOperations"`
	PublishedBatches       uint64 `json:"publishedBatches"`
	PublishedChanges       uint64 `json:"publishedChanges"`
	WatcherDeliveries      uint64 `json:"watcherDeliveries"`
	InitialSnapshots       uint64 `json:"initialSnapshots"`
	QueryRecomputes        uint64 `json:"queryRecomputes"`
	SnapshotsEmitted       uint64 `json:"snapshotsEmitted"`
	DocumentsEmitted       uint64 `json:"documentsEmitted"`
	SlowConsumers          uint64 `json:"slowConsumers"`
}

type ReclaimV2Options 

type ReclaimV2Options struct {
	Online      bool
	MaxAttempts int
	// MemoryOnly skips the physical FreeSpace maintenance generation. It keeps
	// the final installation pause O(1), but another audit is needed after reopen.
	MemoryOnly bool
}

ReclaimV2Options controls explicit page reclamation. Online scans a duplicate read handle without holding the storage writer lock and installs its result only if the Meta generation is unchanged. MaxAttempts bounds complete graph rescans after concurrent commits; zero selects three attempts.

type ReclaimV2Result 

type ReclaimV2Result struct {
	PhysicalPages   uint64
	ReachablePages  uint64
	ReusablePages   uint64
	PinnedSnapshots uint64
	Attempts        int
	Online          bool
	Persisted       bool
}

type ReclamationStats 

type ReclamationStats struct {
	Active          uint64        `json:"active"`
	Attempts        uint64        `json:"attempts"`
	Scans           uint64        `json:"scans"`
	Conflicts       uint64        `json:"conflicts"`
	Completed       uint64        `json:"completed"`
	Failed          uint64        `json:"failed"`
	LastAttempts    uint64        `json:"lastAttempts"`
	LastOnline      bool          `json:"lastOnline"`
	LastReachable   uint64        `json:"lastReachable"`
	LastReclaimable uint64        `json:"lastReclaimable"`
	LastDuration    time.Duration `json:"lastDurationNanos"`
}

type RecoveryMode 

type RecoveryMode uint8

RecoveryMode controls whether Open may perform only the bounded recovery actions described by RecoveryReport. Zero selects the normal automatic mode. 

const (
	RecoveryAutomatic RecoveryMode = iota
	RecoveryRequireClean
)

type RecoveryReport 

type RecoveryReport struct {
	SchemaVersion          int    `json:"schemaVersion"`
	Engine                 string `json:"engine"`
	Created                bool   `json:"created"`
	Recovered              bool   `json:"recovered"`
	CommitSequenceBefore   uint64 `json:"commitSequenceBefore"`
	CommitSequenceAfter    uint64 `json:"commitSequenceAfter"`
	SelectedMetaSlot       uint8  `json:"selectedMetaSlot"`
	ChecksumValidMetaSlots uint8  `json:"checksumValidMetaSlots"`
	RootValidMetaSlots     uint8  `json:"rootValidMetaSlots"`
	MetaRedundancyDegraded bool   `json:"metaRedundancyDegraded"`
	FallbackToOlderRoot    bool   `json:"fallbackToOlderRoot"`
	MainTailBytesRemoved   uint64 `json:"mainTailBytesRemoved"`
	WALRecordsReplayed     uint64 `json:"walRecordsReplayed"`
	WALTailBytesRemoved    uint64 `json:"walTailBytesRemoved"`
	AccelerationDegraded   bool   `json:"accelerationDegraded"`
}

RecoveryReport is an immutable, non-sensitive receipt for decisions made while opening a database. It reports only actions that were completed before Open returned successfully; corruption and unsupported formats still fail Open instead of being described as recovered.

type ReplicationFrame 

type ReplicationFrame struct {
	Type       string
	DatabaseID [16]byte
	AfterToken uint64 // hello: receiver's durable position
	MaxBytes   int    // hello: receiver frame cap
	Batch      *DurableDatabaseChangeBatch
	AckToken   uint64
	Reason     string
}

ReplicationFrame is a transport-neutral protocol envelope. A transport must authenticate both peers (for example with mTLS) before it accepts frames; DatabaseID binds every frame to one durable source identity.

func UnmarshalReplicationFrame 

func UnmarshalReplicationFrame(data []byte, limits ReplicationFrameLimits) (ReplicationFrame, error)

UnmarshalReplicationFrame rejects unknown fields, duplicate JSON keys, malformed base64, invalid typed documents and non-canonical identities before a receiver reaches the follower state machine.

type ReplicationFrameLimits 

type ReplicationFrameLimits struct{ MaxFrameBytes int }

ReplicationFrameLimits bounds one already-decompressed protocol frame. The default accommodates the configured 64 MiB canonical transaction limit plus JSON/base64 overhead, while still rejecting unbounded peer allocation.

type ReplicationSourceLease 

type ReplicationSourceLease struct {
	// contains filtered or unexported fields
}

ReplicationSourceLease gives one authenticated source-side replica identity exclusive process-local ownership of its durable consumer. A lease prevents duplicate concurrent transports from racing one checkpoint; it does not establish distributed primary authority or replace follower-promotion fencing.

func (*ReplicationSourceLease) Release 

func (lease *ReplicationSourceLease) Release()

Release is idempotent. It does not alter the durable consumer checkpoint; normal source-session ACK handling remains the only way to advance that recovery position.

type ReplicationSourceSession 

type ReplicationSourceSession struct {
	// contains filtered or unexported fields
}

ReplicationSourceSession is the primary-side state machine for one authenticated peer. It deliberately permits one unacknowledged batch at a time: this is both bounded flow control and the proof that a durable ACK can never skip an unseen source token.

func NewReplicationSourceSession 

func NewReplicationSourceSession(db *DB, subscription *DurableDatabaseChangeSubscription, limits ReplicationFrameLimits) (*ReplicationSourceSession, error)

NewReplicationSourceSession binds an existing named durable database feed to one source identity. The caller owns peer authentication and must close the session when that authenticated connection ends.

func (*ReplicationSourceSession) AcceptAck 

func (session *ReplicationSourceSession) AcceptAck(frame ReplicationFrame) error

AcceptAck makes one remote acknowledgement durable on the source. The token must exactly match the one batch in flight; stale, future or duplicate ACKs cannot release retained history.

func (*ReplicationSourceSession) AcceptHello 

func (session *ReplicationSourceSession) AcceptHello(frame ReplicationFrame) (*ReplicationFrame, error)

AcceptHello validates a receiver's exact durable position. A different identity or checkpoint is a safe resync response, never an implicit rewind or jump of the source durable consumer.

func (*ReplicationSourceSession) Checkpoint 

func (session *ReplicationSourceSession) Checkpoint() (uint64, error)

func (*ReplicationSourceSession) Close 

func (session *ReplicationSourceSession) Close()

func (*ReplicationSourceSession) NextFrame 

func (session *ReplicationSourceSession) NextFrame(ctx context.Context) (*ReplicationFrame, error)

NextFrame waits for the next ordered source batch. The caller must not send another frame until AcceptAck succeeds for this token. If the peer's declared frame cap cannot carry one atomic Commit Log batch, the session terminates with snapshot_required rather than splitting or partially sending a commit.

type ResourceLimits 

type ResourceLimits struct {
	MaxDocumentBytes         uint64 `json:"maxDocumentBytes"`
	MaxTransactionBytes      uint64 `json:"maxTransactionBytes"`
	MaxTransactionChanges    uint64 `json:"maxTransactionChanges"`
	MaxIndexBuildEntries     uint64 `json:"maxIndexBuildEntries"`
	MaxIndexBuildBytes       uint64 `json:"maxIndexBuildBytes"`
	MaxReactiveViewDocuments uint64 `json:"maxReactiveViewDocuments"`
	MaxReactiveViewBytes     uint64 `json:"maxReactiveViewBytes"`
}

ResourceLimits bounds work admitted by write and index-maintenance APIs. Zero values select production defaults; limits cannot be disabled accidentally. Byte limits use the canonical typed binary representation, independent of Go heap layout, JSON spelling, storage generation, or transport compression.

type ResourceStats 

type ResourceStats struct {
	Limits     ResourceLimits `json:"limits"`
	Rejections uint64         `json:"rejections"`
}

type RollbackAnchor 

type RollbackAnchor struct {
	DatabaseID            [16]byte
	MinimumCommitSequence uint64
	MinimumGeneration     uint64
}

RollbackAnchor is trusted state retained outside the database device. A server must never accept the same identity below either an acknowledged logical commit sequence or physical maintenance generation after restart. The coordinates are independently monotonic: one group may advance several logical sequences while publishing a single physical generation.

type RollbackAnchorStatusProvider 

type RollbackAnchorStatusProvider interface {
	RollbackAnchorStatus() RollbackAnchorStoreStatus
}

RollbackAnchorStatusProvider is an optional lock-free observability contract for RollbackAnchorStore implementations.

type RollbackAnchorStore 

type RollbackAnchorStore interface {
	Load(context.Context) (RollbackAnchor, bool, error)
	Advance(context.Context, RollbackAnchor) error
}

RollbackAnchorStore durably loads and atomically advances one database's monotonic anchor. Advance must not return until the anchor is persistent and must reject identity changes or regression of either monotonic coordinate. Implementations must be safe for concurrent callers and honor cancellation. An Advance error does not prove that state was unchanged: persistence may have completed before a response, deadline or cancellation was observed.

func NewFileRollbackAnchorStore 

func NewFileRollbackAnchorStore(path string) (RollbackAnchorStore, error)

NewFileRollbackAnchorStore returns a fail-closed, atomically replaced anchor file. The parent directory must already exist. For rollback protection, that directory must be backed by storage trusted independently from the database.

type RollbackAnchorStoreStatus 

type RollbackAnchorStoreStatus struct {
	Replicas               uint64 `json:"replicas"`
	Quorum                 uint64 `json:"quorum"`
	Loads                  uint64 `json:"loads"`
	Advances               uint64 `json:"advances"`
	EndpointFailures       uint64 `json:"endpointFailures"`
	QuorumFailures         uint64 `json:"quorumFailures"`
	Conflicts              uint64 `json:"conflicts"`
	AuthenticationFailures uint64 `json:"authenticationFailures"`
	ProtocolFailures       uint64 `json:"protocolFailures"`
	ConfigurationFailures  uint64 `json:"configurationFailures"`
}

RollbackAnchorStoreStatus is a bounded, identity-free process-session view of an anchor backend. Counters are diagnostic and never participate in recovery.

type SortField 

type SortField struct {
	Path      string `json:"path"`
	Direction int    `json:"direction"`
}

type StorageFormat 

type StorageFormat string

StorageFormat identifies the on-disk engine family without opening or mutating the database. Unknown denotes a missing or zero-length path, not an unrecognized non-empty file. 

const (
	StorageFormatUnknown StorageFormat = ""
	StorageFormatV1      StorageFormat = "v1"
	StorageFormatV2      StorageFormat = "v2"
)

func DetectStorageFormat 

func DetectStorageFormat(path string) (StorageFormat, error)

DetectStorageFormat reads only the two fixed meta-page magic fields. The selected engine remains responsible for checksums and complete validation. A non-empty unknown or mixed-family file fails closed as ErrCorrupt.

type StorageFormatInfo 

type StorageFormatInfo struct {
	Format            StorageFormat `json:"format"`
	Revision          uint16        `json:"revision"`
	Generation        uint64        `json:"generation"`
	CommitSequence    uint64        `json:"commitSequence"`
	PhysicalPageCount uint64        `json:"physicalPageCount,omitempty"`
	RequiredFeatures  uint64        `json:"requiredFeatures"`
	OptionalFeatures  uint64        `json:"optionalFeatures"`
	DatabaseIDHex     string        `json:"databaseIdHex,omitempty"`
	ValidMetaSlots    int           `json:"validMetaSlots"`
	ReaderCompatible  bool          `json:"readerCompatible"`
}

StorageFormatInfo is a read-only negotiation view, not a full graph audit. ReaderCompatible says this binary understands the reported revision and all required feature bits; callers must still Open the database before use.

func InspectStorageFormat 

func InspectStorageFormat(path string) (StorageFormatInfo, error)

InspectStorageFormat validates stable Meta checksums and reports the newest negotiation envelope without locking, opening, migrating, or mutating the database. For checksum-valid future V2 revisions it still reports revision and feature bits while ReaderCompatible is false.

type StorageStats 

type StorageStats struct {
	Engine                     string                    `json:"engine"`
	RollbackProtected          bool                      `json:"rollbackProtected"`
	RollbackAnchorSequence     uint64                    `json:"rollbackAnchorSequence"`
	RollbackAnchorGeneration   uint64                    `json:"rollbackAnchorGeneration"`
	RollbackAnchorFailures     uint64                    `json:"rollbackAnchorFailures"`
	RollbackAnchorTimeout      time.Duration             `json:"rollbackAnchorTimeoutNanos"`
	RollbackAnchorNanos        uint64                    `json:"rollbackAnchorNanos"`
	RollbackAnchorMaxLatency   time.Duration             `json:"rollbackAnchorMaxLatencyNanos"`
	RollbackAnchorStore        RollbackAnchorStoreStatus `json:"rollbackAnchorStore"`
	PageSize                   uint64                    `json:"pageSize"`
	Generation                 uint64                    `json:"generation"`
	PhysicalPages              uint64                    `json:"physicalPages"`
	CommitSequence             uint64                    `json:"commitSequence"`
	OldestRetainedSequence     uint64                    `json:"oldestRetainedSequence"`
	RetainedCommits            uint64                    `json:"retainedCommits"`
	CommitRetentionMax         uint64                    `json:"commitRetentionMax"`
	CommitRetentionOverage     uint64                    `json:"commitRetentionOverage"`
	RetainedCommitBytes        uint64                    `json:"retainedCommitBytes"`
	CommitRetentionMaxBytes    uint64                    `json:"commitRetentionMaxBytes"`
	CommitRetentionByteOverage uint64                    `json:"commitRetentionByteOverage"`
	RetentionPrunedCommits     uint64                    `json:"retentionPrunedCommits"`
	RetentionPressureEvents    uint64                    `json:"retentionPressureEvents"`
	RetentionPressure          bool                      `json:"retentionPressure"`
	StorageUsedBytes           uint64                    `json:"storageUsedBytes"`
	StorageMaxBytes            uint64                    `json:"storageMaxBytes"`
	StorageByteOverage         uint64                    `json:"storageByteOverage"`
	StorageLimitRejections     uint64                    `json:"storageLimitRejections"`
	StorageQuotaExhausted      bool                      `json:"storageQuotaExhausted"`
	ActiveReaders              uint64                    `json:"activeReaders"`
	ActiveReplayLeases         uint64                    `json:"activeReplayLeases"`
	Documents                  uint64                    `json:"documents"`
	Collections                uint64                    `json:"collections"`
	ReusablePages              uint64                    `json:"reusablePages"`
	TreeSplits                 uint64                    `json:"treeSplits"`
	TreeMerges                 uint64                    `json:"treeMerges"`
	PersistentFreeSpace        bool                      `json:"persistentFreeSpace"`
	FreeSpaceLoads             uint64                    `json:"freeSpaceLoads"`
	FreeSpaceLoadFailures      uint64                    `json:"freeSpaceLoadFailures"`
	FreeSpacePublishes         uint64                    `json:"freeSpacePublishes"`
	FreeSpaceCandidateChecks   uint64                    `json:"freeSpaceCandidateChecks"`
	PageCache                  PageCacheStats            `json:"pageCache"`
	DocumentCache              DocumentCacheStats        `json:"documentCache"`
	CommitAttempts             uint64                    `json:"commitAttempts"`
	CommittedTransactions      uint64                    `json:"committedTransactions"`
	RejectedTransactions       uint64                    `json:"rejectedTransactions"`
	CommitNanos                uint64                    `json:"commitNanos"`
	CommitMaxLatency           time.Duration             `json:"commitMaxLatencyNanos"`
}

StorageStats describes the selected physical backend. Session counters reset on reopen; physical state and cache counters come from the backend itself.

type Update 

type Update map[string]any

type UpdateResult 

type UpdateResult struct{ MatchedCount, ModifiedCount int64 }

type V1CheckpointPolicy 

type V1CheckpointPolicy struct {
	MaxWALBytes   int64
	MaxWALCommits uint64
	Disabled      bool
}

V1CheckpointPolicy bounds the legacy sidecar WAL. Either enabled threshold triggers a synchronous physical checkpoint after the triggering logical commit is already durable. Zero values select production defaults.

type V1Options 

type V1Options struct {
	Checkpoint     V1CheckpointPolicy
	Recovery       RecoveryMode
	ResourceLimits ResourceLimits
}

V1Options configures the explicitly selected legacy V1 storage engine. Storage V2 does not use this policy because every V2 commit publishes a COW database root and inactive Meta page atomically.

type V2CommitCoordinatorOptions 

type V2CommitCoordinatorOptions struct {
	Enabled    bool
	MaxBatch   int
	MaxPending int
	MaxDelay   time.Duration
}

V2CommitCoordinatorOptions controls optional group commit for ordinary V2 InsertMany, filter Update and filter Delete operations. It is disabled by default, so opening an existing database never changes write scheduling unexpectedly.

A coordinator group has one physical V2 Meta publication but retains one logical commit token for every admitted write request. Public write transactions, atomic RPC, index builds and other maintenance operations remain exclusive commits. When rollback protection is configured, the coordinator advances the external anchor only after the group's final Meta publication is durable and before acknowledging any member.

type V2CommitCoordinatorStats 

type V2CommitCoordinatorStats struct {
	Enabled             bool   `json:"enabled"`
	Pending             uint64 `json:"pending"`
	PendingCapacity     uint64 `json:"pendingCapacity"`
	Admitted            uint64 `json:"admitted"`
	AdmissionRejected   uint64 `json:"admissionRejected"`
	Batches             uint64 `json:"batches"`
	GroupedTransactions uint64 `json:"groupedTransactions"`
	OutcomeUnknown      uint64 `json:"outcomeUnknown"`
}

V2CommitCoordinatorStats is a fixed-cardinality snapshot of the optional V2 write-admission scheduler. It is included in DBStats and the versioned admin schema, so applications can alert on admission pressure without inspecting a mutable queue or adding application labels.

type V2CommitRetentionPolicy 

type V2CommitRetentionPolicy struct {
	MaxCommits uint64
	MaxBytes   uint64
}

V2CommitRetentionPolicy bounds logical Commit Log history by both commit count and canonical encoded bytes. Zero fields select production defaults. Active replay leases may temporarily exceed either budget rather than losing history under a reader.

type V2DestinationOptions 

type V2DestinationOptions struct {
	StorageLimits  V2StorageLimits
	ResourceLimits ResourceLimits
}

V2DestinationOptions configures newly written migration or compaction files. ResourceLimits govern transient index construction as well as the reopened destination handle; zero fields select production defaults.

type V2Follower 

type V2Follower struct {
	// contains filtered or unexported fields
}

V2Follower owns a local, read-only V2 database that advances only through validated DurableDatabaseChangeBatch values. It is the local application half of a future remote replication protocol; transport authentication, snapshot transfer and promotion deliberately remain outside this type.

func OpenV2Follower 

func OpenV2Follower(path string, options V2Options) (*V2Follower, error)

OpenV2Follower opens a physical archive/bootstrap copy as a replica. Normal public mutations on DB return ErrReplicaReadOnly; use Apply to advance the next source token. The returned DB remains fully queryable and reactive.

func (*V2Follower) Apply 

func (follower *V2Follower) Apply(ctx context.Context, source DurableDatabaseChangeBatch) error

Apply durably and atomically applies exactly the next source batch. A gap, duplicate or locally diverged token returns ErrReplicaSequence; it never guesses, retries business logic or advances past missing history.

func (*V2Follower) ApplyFrame 

func (follower *V2Follower) ApplyFrame(ctx context.Context, frame ReplicationFrame) error

ApplyFrame binds the decoded transport envelope to this bootstrap's durable identity before applying its batch. A transport handles hello/ack/resync; only a validated batch frame belongs at the follower mutation boundary.

func (*V2Follower) Close 

func (follower *V2Follower) Close() error

func (*V2Follower) DB 

func (follower *V2Follower) DB() *DB

func (*V2Follower) Promote 

func (follower *V2Follower) Promote(ctx context.Context, authority FollowerPromotionAuthority) (FollowerPromotionFence, error)

Promote makes this follower writable only after an external authority fences the former primary at this exact identity/token. There is intentionally no default or best-effort implementation: promoting without a durable external fence would turn a network partition into split-brain data loss.

type V2Maintenance 

type V2Maintenance struct {
	// contains filtered or unexported fields
}

V2Maintenance owns one background reclamation loop. Stop is idempotent and waits for an active scan to observe cancellation. Closing the DB also stops the loop through the DB lifecycle channel.

func (*V2Maintenance) Done 

func (maintenance *V2Maintenance) Done() <-chan struct{}

func (*V2Maintenance) Stats 

func (maintenance *V2Maintenance) Stats() V2MaintenanceStats

func (*V2Maintenance) Stop 

func (maintenance *V2Maintenance) Stop()

type V2MaintenanceOptions 

type V2MaintenanceOptions struct {
	Interval       time.Duration
	Timeout        time.Duration
	MaxAttempts    int
	RunImmediately bool
	// PersistFreeSpace opts into a physical maintenance generation after each
	// successful scan. The default memory-only mode minimizes writer pauses.
	PersistFreeSpace bool
}

V2MaintenanceOptions configures an explicit default-off maintenance loop. Every run uses online optimistic reclamation; runs never overlap.

type V2MaintenanceStats 

type V2MaintenanceStats struct {
	Runs         uint64
	Completed    uint64
	Conflicts    uint64
	Failed       uint64
	Active       bool
	LastDuration time.Duration
	LastError    string
}

type V2Options 

type V2Options struct {
	Recovery              RecoveryMode
	CommitRetention       V2CommitRetentionPolicy
	ReplayDeliveryTimeout time.Duration
	CommitCoordinator     V2CommitCoordinatorOptions
	ResourceLimits        ResourceLimits
	StorageLimits         V2StorageLimits
	RollbackProtection    V2RollbackProtection
	// RequireGraphAudit rejects a V2 database at startup when any page
	// protected by the current or fallback Meta root is structurally invalid.
	// It is intentionally opt-in because audit cost grows with database size.
	// This does not replace the offline semantic index verifier.
	RequireGraphAudit bool
	// RequirePrivateFileMode rejects a V2 file with group/world permission
	// bits instead of silently changing operator-owned permissions.
	RequirePrivateFileMode bool
	// PrimaryWriteFence optionally proves that this local database still holds
	// external primary authority before every business V2 commit. It is not
	// consulted by a read-only follower applying an already validated source
	// batch. The guard must be local, non-blocking and safe for concurrent use;
	// controller I/O/lease renewal belongs outside Meldbase's writer lock.
	PrimaryWriteFence V2PrimaryWriteFence
	// Follower marks this local open as a replica. Normal application writes
	// fail with ErrReplicaReadOnly; only V2Follower.Apply may advance it.
	Follower bool
}

V2Options configures explicitly selected Storage V2 opening.

type V2PrimaryWriteFence 

type V2PrimaryWriteFence interface {
	ValidateV2PrimaryWrite(PrimaryWriteFenceRequest) error
}

V2PrimaryWriteFence is the local enforcement hook for an external primary election/fencing system. Its implementation normally checks an atomically refreshed lease epoch and expiry, not the network. Returning an error rejects the whole logical commit before V2 storage mutation; it never poisons the database or advances a token.

Implementations must not call back into DB and must return promptly: the check runs while the V2 writer has admitted a commit. Election, renewal, certificate rotation and old-primary revocation remain external concerns.

type V2PrimaryWriteFenceStats 

type V2PrimaryWriteFenceStats struct {
	Configured bool   `json:"configured"`
	Enforced   bool   `json:"enforced"`
	Checks     uint64 `json:"checks"`
	Rejected   uint64 `json:"rejected"`
}

V2PrimaryWriteFenceStats is a fixed-cardinality view of the optional external primary-write guard. Configured means a guard was supplied at open; Enforced is false while a read-only follower applies validated source history. Checks and Rejected count only actual primary write admissions. No lease, epoch, endpoint, database ID, or controller detail is exposed.

type V2RollbackProtection 

type V2RollbackProtection struct {
	ExpectedDatabaseID    [16]byte
	MinimumCommitSequence uint64
	MinimumGeneration     uint64
	AnchorStore           RollbackAnchorStore
	InitializeAnchor      bool
	// OperationTimeout bounds each Load/Advance/read-back interaction. Zero
	// selects DefaultRollbackAnchorOperationTimeout.
	OperationTimeout time.Duration
}

V2RollbackProtection configures fail-closed database identity and sequence checks. AnchorStore should live on an independently trusted device or remote quorum; placing it beside the database cannot detect whole-device rollback. InitializeAnchor explicitly trusts the database currently at Path when the store is empty and should only be used during provisioning or audited restore.

type V2StorageLimits 

type V2StorageLimits struct{ MaxFileBytes uint64 }

V2StorageLimits bounds the physical single-file high-water mark. Zero selects DefaultV2MaxFileBytes. The value must be a 16 KiB V2 page multiple.

type V2VerificationReport 

type V2VerificationReport struct {
	SchemaVersion              int           `json:"schemaVersion"`
	Verified                   bool          `json:"verified"`
	Format                     StorageFormat `json:"format"`
	Revision                   uint16        `json:"revision"`
	DatabaseIDHex              string        `json:"databaseIdHex"`
	MetaGeneration             uint64        `json:"metaGeneration"`
	CommitSequence             uint64        `json:"commitSequence"`
	OldestRetainedSequence     uint64        `json:"oldestRetainedSequence"`
	RequiredFeatures           uint64        `json:"requiredFeatures"`
	OptionalFeatures           uint64        `json:"optionalFeatures"`
	ValidMetaSlots             int           `json:"validMetaSlots"`
	FileBytes                  uint64        `json:"fileBytes"`
	TrailingBytes              uint64        `json:"trailingBytes"`
	PhysicalPages              uint64        `json:"physicalPages"`
	CommittedPhysicalPages     uint64        `json:"committedPhysicalPages"`
	ReachablePages             uint64        `json:"reachablePages"`
	ReclaimablePages           uint64        `json:"reclaimablePages"`
	PersistentFreeSpace        bool          `json:"persistentFreeSpace"`
	FreeSpaceValid             bool          `json:"freeSpaceValid"`
	IndexContentsVerified      bool          `json:"indexContentsVerified"`
	IndexBuildContentsVerified bool          `json:"indexBuildContentsVerified"`
	SHA256                     string        `json:"sha256"`
}

V2VerificationReport is a schema-versioned receipt for a full, read-only V2 protected-page graph and published-index semantic audit. ReclaimablePages is informational; verification never installs a free pool or publishes maintenance metadata.

func VerifyV2File 

func VerifyV2File(ctx context.Context, path string) (V2VerificationReport, error)

VerifyV2File performs an offline, non-mutating audit of an existing V2 file. It takes a non-blocking shared advisory lock, so an active writer fails with ErrDatabaseLocked. It never creates, truncates, repairs, reclaims, or advances the database. Meta inspection alone is cheaper; this method walks every page protected by both valid Meta roots, recomputes published and provable shadow Secondary keys from canonical Primary documents in both directions, and hashes the file. Legacy caught-up builds lacking an applied CatalogRoot remain readable but report IndexBuildContentsVerified=false.

type Value 

type Value struct {
	// contains filtered or unexported fields
}

Value is a closed tagged value. Its representation is private so callers cannot construct a tag/payload mismatch.

func Array 

func Array(v ...Value) Value

func Binary 

func Binary(v []byte) Value

func Bool 

func Bool(v bool) Value

func Float 

func Float(v float64) Value

func ID 

func ID(v DocumentID) Value

func Int 

func Int(v int64) Value

func Null 

func Null() Value

func Object 

func Object(v Document) Value

func String 

func String(v string) Value

func Time 

func Time(v time.Time) Value

Time stores millisecond precision, matching JavaScript Date and the wire contract. Precision is normalized at construction rather than silently lost during transport.

func UnmarshalWireValue 

func UnmarshalWireValue(data []byte, limits QueryLimits) (Value, error)

UnmarshalWireValue decodes one closed, typed wire value using the same depth/item/byte limits as query operands. It is suitable for data-only protocol arguments such as RPC; it never evaluates source or callbacks.

func ValueOf 

func ValueOf(x any) (Value, error)

func (Value) ArrayValue 

func (v Value) ArrayValue() ([]Value, bool)

func (Value) BinaryValue 

func (v Value) BinaryValue() ([]byte, bool)

func (Value) Bool 

func (v Value) Bool() (bool, bool)

func (Value) Clone 

func (v Value) Clone() Value

func (Value) Equal 

func (v Value) Equal(other Value) bool

func (Value) Float64 

func (v Value) Float64() (float64, bool)

func (Value) IDValue 

func (v Value) IDValue() (DocumentID, bool)

func (Value) Int64 

func (v Value) Int64() (int64, bool)

func (Value) Kind 

func (v Value) Kind() Kind

func (Value) ObjectValue 

func (v Value) ObjectValue() (Document, bool)

func (Value) StringValue 

func (v Value) StringValue() (string, bool)

func (Value) TimeValue 

func (v Value) TimeValue() (time.Time, bool)

type WriteTransaction 

type WriteTransaction struct {
	// contains filtered or unexported fields
}

WriteTransaction is a short-lived snapshot write view. It provides point operations with optimistic serializable commit validation. Values returned from it are isolated clones.

A transaction is active only during its handler callback. Handlers must not retain it or call normal DB/Collection methods from inside the callback.

func (*WriteTransaction) DeleteOne 

func (tx *WriteTransaction) DeleteOne(collection string, id DocumentID) error

DeleteOne stages a point delete. Deleting a document inserted earlier in the same callback cancels that insert without producing a storage mutation.

func (*WriteTransaction) Find 

func (tx *WriteTransaction) Find(collection string, query QuerySpec) (QuerySnapshot, error)

Find evaluates query against the transaction's immutable V2 snapshot plus its own staged point writes. It installs a collection snapshot fence, so any concurrent document or published-index change in that collection causes the eventual commit to return ErrWriteConflict rather than admitting a phantom.

The first range-read primitive intentionally uses a collection-wide fence. It is serializable and independent of a process-local index plan; a future narrower predicate-fence implementation can refine conflicts without weakening this API's correctness contract.

func (*WriteTransaction) GetOne 

func (tx *WriteTransaction) GetOne(collection string, id DocumentID) (Document, error)

GetOne returns one document by intrinsic ID from the transaction's current view, including earlier point mutations in the same callback.

func (*WriteTransaction) InsertOne 

func (tx *WriteTransaction) InsertOne(collection string, document Document) (DocumentID, error)

InsertOne stages one insert and returns its generated or supplied ID.

func (*WriteTransaction) MeldbaseStageSystemMutation 

func (tx *WriteTransaction) MeldbaseStageSystemMutation(mutation systemrecord.Mutation, onCommit func(uint64)) error

MeldbaseStageSystemMutation is first-party composite-transaction plumbing. Its internal parameter type prevents external applications from accessing the private System tree. onCommit runs synchronously after the durable root is published and before the matching business ChangeBatch becomes visible; it must be bounded, non-blocking and must not call back into the database.

func (*WriteTransaction) ReplaceOne 

func (tx *WriteTransaction) ReplaceOne(collection string, id DocumentID, document Document) error

ReplaceOne stages a full replacement for an existing document. The supplied document may omit _id; if present it must equal id.

func (*WriteTransaction) UpdateOne 

func (tx *WriteTransaction) UpdateOne(collection string, id DocumentID, mutation MutationSpec) error

UpdateOne applies one already compiled, data-only mutation to the transaction's current document view. Earlier writes in the same transaction are visible and the intrinsic _id remains immutable.

type WriteTransactionStats 

type WriteTransactionStats struct {
	Active    uint64 `json:"active"`
	Started   uint64 `json:"started"`
	Committed uint64 `json:"committed"`
	Noops     uint64 `json:"noops"`
	Conflicts uint64 `json:"conflicts"`
	Aborted   uint64 `json:"aborted"`
}

WriteTransactionStats describes public optimistic point transactions. Every started callback reaches exactly one terminal counter. These aggregates do not contain collection, document, principal, or callback identifiers.

Source Files

View all Source files

Directories

Path Synopsis
Package admin provides optional, bounded observability consumers for Meldbase.
 Package admin provides optional, bounded observability consumers for Meldbase.
cmd
meld command
meld-power-redfish-adapter command
integrations
anchorhttp
Package anchorhttp provides an authenticated HTTPS quorum implementation of meldbase.RollbackAnchorStore.
 Package anchorhttp provides an authenticated HTTPS quorum implementation of meldbase.RollbackAnchorStore.
authorityhttp
Package authorityhttp exposes a narrow, mTLS-authenticated primary-lease Authority control endpoint.
 Package authorityhttp exposes a narrow, mTLS-authenticated primary-lease Authority control endpoint.
leasehttp
Package leasehttp exposes one authenticated primarylease.LeaseStore member over strict HTTPS/mTLS JSON.
 Package leasehttp exposes one authenticated primarylease.LeaseStore member over strict HTTPS/mTLS JSON.
otel
Package meldotel exports Meldbase's bounded admin snapshots through the stable OpenTelemetry Metrics API.
 Package meldotel exports Meldbase's bounded admin snapshots through the stable OpenTelemetry Metrics API.
primarylease
Package primarylease supplies a locally verifiable, short-lived primary write lease for Meldbase V2 deployments.
 Package primarylease supplies a locally verifiable, short-lived primary write lease for Meldbase V2 deployments.
replicationauth
Package replicationauth provides shared identity primitives for trusted server-to-server replication transports.
 Package replicationauth provides shared identity primitives for trusted server-to-server replication transports.
replicationhttp
Package replicationhttp transports a verified V2 bootstrap over HTTPS.
 Package replicationhttp transports a verified V2 bootstrap over HTTPS.
replicationws
Package replicationws adapts Meldbase's trusted-server replication protocol to WebSocket.
 Package replicationws adapts Meldbase's trusted-server replication protocol to WebSocket.
internal
index
policyrecord
Package policyrecord defines the durable private representation of server query-policy generations.
 Package policyrecord defines the durable private representation of server query-policy generations.
qualification
Package qualification contains operational release-evidence runners.
 Package qualification contains operational release-evidence runners.
server
storage
storage/v2
Package v2 contains the experimental Meldbase Storage V2 page format.
 Package v2 contains the experimental Meldbase Storage V2 page format.
systemrecord
Package systemrecord defines the private bridge between the root database and higher-level built-in services.
 Package systemrecord defines the private bridge between the root database and higher-level built-in services.
wal
Package server exposes Meldbase's authenticated HTTP, WebSocket realtime and data-only RPC transport.
 Package server exposes Meldbase's authenticated HTTP, WebSocket realtime and data-only RPC transport.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.