meldbase

package module
v0.1.0-alpha.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 15, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

Meldbase

Documents that stay live. Local by design.

Meldbase is an experimental, embedded reactive document database written in Go. It combines a typed document model, local durable storage, query planning, and live query subscriptions behind one coherent API. It is a new database—not a MongoDB protocol or behavior clone.

The current implementation contains a typed Go engine, crash-recoverable copy-on-write checkpoints with a redo WAL, scalar B+Tree indexes, a shared Go/TypeScript query contract, and a secured HTTP/WebSocket reactive server.

It is still early-stage: normal operation uses a .wal sidecar, realtime V1 recomputes full snapshots instead of sending incremental diffs, B+Tree deletion still rebuilds instead of locally rebalancing, and the property/crash/benchmark matrix is incomplete. “One file” remains a product target, not a claim about this build.

Installation

The Go module is published from:

go get github.com/crapthings/meldbase@latest

The TypeScript packages currently live in this pnpm workspace and are not yet published to npm. Use the repository workspace for the developer preview; do not assume @meldbase/client or @meldbase/react is available from the public npm registry until an npm release is announced.

Go quick start

db, err := meldbase.Open("app.meld")
if err != nil { log.Fatal(err) }
defer db.Close()

users := db.Collection("users")
id, err := users.InsertOne(ctx, meldbase.Document{
  "name": meldbase.String("Ada"),
  "age":  meldbase.Int(30),
})

err = users.CreateIndex(ctx, "users_age", []meldbase.IndexField{
  {Field: "age", Order: 1},
}, meldbase.IndexOptions{})

TypeScript SDK

The same compiled, data-only query is used by the local cache and sent to the server. Native JS applications can use a local reactive collection directly:

import { LocalCollection } from "@meldbase/client"

const todos = new LocalCollection([
  { _id: "one", title: "Learn Meldbase", done: false }
])

const open = todos.find({ done: false }, {
  sort: [{ path: "title", direction: 1 }]
})

const stop = open.subscribe(snapshot => render(snapshot))

Remote queries use HTTP for fetch and a ticket-authenticated WebSocket for their ongoing state:

import { MeldbaseClient } from "@meldbase/client"

const db = new MeldbaseClient({
  baseUrl: "https://data.example.com",
  accessToken: () => auth.currentAccessToken()
})

const query = db.collection("todos").find({ done: false })
const stop = query.subscribe(render, {
  onStatus: status => showSyncState(status.state)
})

const created = await db.collection("todos").insertOne({
  title: "Build something live",
  done: false
})

await db.collection("todos").updateOne(
  { _id: created._id },
  { $set: { done: true } }
)

React uses a thin useSyncExternalStore adapter over that same query object; it does not introduce a second query language:

import { useMemo } from "react"
import { useLiveQuery } from "@meldbase/react"

function OpenTodos({ db }: { db: MeldbaseClient }) {
  const query = useMemo(
    () => db.collection("todos").find({ done: false }),
    [db]
  )
  const { documents, status, error } = useLiveQuery(query)
  // Keep the query object stable; updates arrive over its WebSocket subscription.
  return <TodoList todos={documents} syncState={status} error={error} />
}

See docs/client-protocol.md for the realtime and security model.

A complete browser example lives in examples/realtime-todos. Run the development server above, then:

pnpm --filter @meldbase/example-realtime-todos dev

The example performs real HTTP mutations and WebSocket snapshots through the React adapter. Open it twice to observe the same query update in both views.

Run the end-to-end demo

The demo performs durable insert/update, creates and uses an index, observes a reactive query, closes the database, and proves the data after reopen:

go run ./cmd/meld demo

Run the HTTP/WebSocket server locally only with the explicit development-auth switch:

go run ./cmd/meld serve \
  --db ./app.meld \
  --addr :8080 \
  --dev-no-auth

--dev-no-auth grants every request full access and is intentionally required; it is not a production authentication mode. A production embedding supplies the server Authenticator and Authorizer implementations itself.

The implemented transport endpoints are:

GET  /health
POST /v1/collections/{collection}/query
POST /v1/collections/{collection}/documents
POST /v1/collections/{collection}/mutations
POST /v1/realtime/tickets
GET  /v1/realtime

HTTP queries carry the same versioned, data-only AST used by the SDK. With the development server above:

curl -X POST http://localhost:8080/v1/collections/todos/query \
  -H 'Content-Type: application/json' \
  --data '{
    "version": 1,
    "query": {
      "version": 1,
      "where": {"op":"compare","cmp":"eq","path":"done","value":{"t":"bool","v":false}},
      "sort": [{"path":"title","direction":1}]
    }
  }'

Browser realtime authentication is two-step: obtain a short-lived, single-use ticket over authenticated HTTP, then send it in the first WebSocket message. Credentials never appear in the WebSocket URL. The core V1 exchange is:

{"v":1,"type":"authenticate","ticket":"<single-use-ticket>"}
{"v":1,"type":"subscribe","requestId":"open-todos","collection":"todos","query":{"version":1,"where":{"op":"compare","cmp":"eq","path":"done","value":{"t":"bool","v":false}}}}
{"v":1,"type":"snapshot","requestId":"open-todos","subscriptionId":"<id>","token":"<signed-resume-token>","documents":[]}
{"v":1,"type":"unsubscribe","subscriptionId":"<id>"}

See docs/client-protocol.md for reconnect, resync_required, limits, origin checks, and row/field authorization.

Status

Early-stage and not suitable for production data. See docs/architecture.md and docs/roadmap.md. The first-stage requirement-to-evidence map is in docs/mvp-audit.md.

Supported query operators are $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $exists, $and, $or, and $not. Meldbase defines these semantics itself and does not promise MongoDB compatibility.

Development

go test ./...
go test -race ./...
go vet ./...
go run ./cmd/meld demo
pnpm check
pnpm test
pnpm build:example

Contributions should follow CONTRIBUTING.md. Security reports must use the private process described in SECURITY.md, not a public issue. Maintainer release gates are documented in docs/releasing.md.

License

Licensed under the Apache License 2.0.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrClosed            = errors.New("meldbase: database is closed")
	ErrInvalidDocument   = errors.New("meldbase: invalid document")
	ErrInvalidFilter     = errors.New("meldbase: invalid filter")
	ErrInvalidUpdate     = errors.New("meldbase: invalid update")
	ErrMutationLimit     = errors.New("meldbase: mutation affected-row limit exceeded")
	ErrNotFound          = errors.New("meldbase: document not found")
	ErrDuplicateID       = errors.New("meldbase: duplicate document id")
	ErrInvalidCollection = errors.New("meldbase: invalid collection")
	ErrImmutableID       = errors.New("meldbase: _id is immutable")
	ErrSlowConsumer      = errors.New("meldbase: change consumer is too slow")
	ErrCorrupt           = errors.New("meldbase: corrupt database")
	ErrDuplicateKey      = errors.New("meldbase: duplicate index key")
	ErrInvalidIndex      = errors.New("meldbase: invalid index")
	ErrDurability        = errors.New("meldbase: durability failure; writes are disabled")
)
View Source 
var DefaultQueryLimits = QueryLimits{
	MaxWireBytes: 1 << 20, MaxDepth: 16, MaxNodes: 128,
	MaxArrayItems: 256, MaxValueBytes: 16_384, MaxSortFields: 4,
	MaxLimit: 10_000,
}

Functions

func MarshalQuerySpecJSON 

func MarshalQuerySpecJSON(query QuerySpec) ([]byte, error)

MarshalQuerySpecJSON emits the canonical, data-only wire representation used for transport fingerprints and cross-language conformance.

func MarshalWireDocument 

func MarshalWireDocument(document Document) ([]byte, error)

func MarshalWireValue 

func MarshalWireValue(value Value) ([]byte, error)

func ValidateStrictJSON 

func ValidateStrictJSON(data []byte, maxBytes int) error

ValidateStrictJSON rejects oversized, trailing, deeply nested, and duplicate-key JSON before a transport decodes it into structs or maps.

Types

type Change 

type Change struct {
	Collection string
	Operation  Operation
	DocumentID DocumentID
	Before     *Document
	After      *Document
	Index      *IndexDefinition
}

type ChangeBatch 

type ChangeBatch struct {
	Token   uint64
	Changes []Change
}

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

func (*Collection) CreateIndex 

func (c *Collection) CreateIndex(ctx context.Context, name string, fields []IndexField, options IndexOptions) error

func (*Collection) DeleteMany 

func (c *Collection) DeleteMany(ctx context.Context, filter Filter) (DeleteResult, error)

func (*Collection) DeleteManyQuery 

func (c *Collection) DeleteManyQuery(ctx context.Context, query QuerySpec) (DeleteResult, error)

func (*Collection) DeleteManyQueryLimited 

func (c *Collection) DeleteManyQueryLimited(ctx context.Context, query QuerySpec, maxAffected int) (DeleteResult, error)

DeleteManyQueryLimited atomically rejects the whole mutation when more than maxAffected documents match.

func (*Collection) DeleteOne 

func (c *Collection) DeleteOne(ctx context.Context, filter Filter) (DeleteResult, error)

func (*Collection) DeleteOneQuery 

func (c *Collection) DeleteOneQuery(ctx context.Context, query QuerySpec) (DeleteResult, error)

func (*Collection) Explain 

func (c *Collection) Explain(ctx context.Context, filter Filter) (ExplainResult, error)

func (*Collection) Find 

func (c *Collection) Find(ctx context.Context, filter Filter, options ...QueryOptions) (*Cursor, error)

func (*Collection) FindOne 

func (c *Collection) FindOne(ctx context.Context, filter Filter) (Document, error)

func (*Collection) FindQuery 

func (c *Collection) FindQuery(ctx context.Context, query QuerySpec) (*Cursor, error)

func (*Collection) InsertMany 

func (c *Collection) InsertMany(ctx context.Context, documents []Document) ([]DocumentID, error)

InsertMany validates IDs, documents, and all unique-index keys before writing one WAL record. Either the entire batch becomes visible or none of it does.

func (*Collection) InsertOne 

func (c *Collection) InsertOne(ctx context.Context, document Document) (DocumentID, error)

func (*Collection) SnapshotQuery 

func (c *Collection) SnapshotQuery(ctx context.Context, query QuerySpec) (QuerySnapshot, error)

func (*Collection) SubscribeQuery 

func (c *Collection) SubscribeQuery(ctx context.Context, query QuerySpec, buffer int) (*QuerySubscription, error)

func (*Collection) UpdateMany 

func (c *Collection) UpdateMany(ctx context.Context, filter Filter, update Update) (UpdateResult, error)

func (*Collection) UpdateManyQuery 

func (c *Collection) UpdateManyQuery(ctx context.Context, query QuerySpec, mutation MutationSpec) (UpdateResult, error)

func (*Collection) UpdateManyQueryLimited 

func (c *Collection) UpdateManyQueryLimited(ctx context.Context, query QuerySpec, mutation MutationSpec, maxAffected int) (UpdateResult, error)

UpdateManyQueryLimited atomically rejects the whole mutation when more than maxAffected documents match. A non-positive limit is invalid here so callers cannot accidentally disable a server-owned safety bound.

func (*Collection) UpdateOne 

func (c *Collection) UpdateOne(ctx context.Context, filter Filter, update Update) (UpdateResult, error)

func (*Collection) UpdateOneQuery 

func (c *Collection) UpdateOneQuery(ctx context.Context, query QuerySpec, mutation MutationSpec) (UpdateResult, error)

type Cursor 

type Cursor struct {
	// contains filtered or unexported fields
}

func (*Cursor) All 

func (c *Cursor) All(ctx context.Context) ([]Document, error)

func (*Cursor) Next 

func (c *Cursor) Next(ctx context.Context) (Document, bool, error)

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func New 

func New() *DB

func Open 

func Open(path string) (*DB, error)

func (*DB) CanResumeFrom 

func (db *DB) CanResumeFrom(token uint64) bool

func (*DB) Close 

func (db *DB) Close() error

func (*DB) Collection 

func (db *DB) Collection(name string) *Collection

func (*DB) DatabaseIdentity 

func (db *DB) DatabaseIdentity() [16]byte

func (*DB) Sync 

func (db *DB) Sync() error

func (*DB) WatchChanges 

func (db *DB) WatchChanges(ctx context.Context, collection string, buffer int) (<-chan ChangeBatch, <-chan error, error)

type DeleteResult 

type DeleteResult struct{ DeletedCount int64 }

type Document 

type Document map[string]Value

func NewDocument 

func NewDocument(fields map[string]any) (Document, error)

func UnmarshalWireDocument 

func UnmarshalWireDocument(data []byte, limits QueryLimits) (Document, error)

func UnmarshalWireInputDocument 

func UnmarshalWireInputDocument(data []byte, limits QueryLimits) (Document, error)

func (Document) Clone 

func (d Document) Clone() Document

func (Document) Equal 

func (d Document) Equal(other Document) bool

func (Document) ID 

func (d Document) ID() (DocumentID, bool)

func (Document) Validate 

func (d Document) Validate() error

Validate checks every nested field and value before a document crosses a storage or transport boundary.

type DocumentID 

type DocumentID [16]byte

func NewDocumentID 

func NewDocumentID() (DocumentID, error)

func ParseDocumentID 

func ParseDocumentID(s string) (DocumentID, error)

func (DocumentID) IsZero 

func (id DocumentID) IsZero() bool

func (DocumentID) String 

func (id DocumentID) String() string

type ExplainResult 

type ExplainResult struct {
	Stage, IndexName                string
	DocumentsExamined, KeysExamined int64
}

type Filter 

type Filter map[string]any

type IndexDefinition 

type IndexDefinition struct {
	Name, Field string
	Order       int
	Unique      bool
}

type IndexField 

type IndexField struct {
	Field string
	Order int
}

type IndexOptions 

type IndexOptions struct{ Unique bool }

type Kind 

type Kind uint8
const (
	NullKind Kind = iota
	BoolKind
	Int64Kind
	Float64Kind
	StringKind
	BinaryKind
	TimeKind
	ArrayKind
	ObjectKind
	IDKind
)

type MutationSpec 

type MutationSpec struct {
	// contains filtered or unexported fields
}

func CompileUpdate 

func CompileUpdate(update Update) (MutationSpec, error)

func DecodeMutationSpecJSON 

func DecodeMutationSpecJSON(data []byte, limits QueryLimits) (MutationSpec, error)

func (MutationSpec) Apply 

func (m MutationSpec) Apply(document Document) (Document, error)

func (MutationSpec) Paths 

func (m MutationSpec) Paths() []string

type Operation 

type Operation string
const (
	InsertOperation      Operation = "insert"
	UpdateOperation      Operation = "update"
	DeleteOperation      Operation = "delete"
	CreateIndexOperation Operation = "create_index"
)

type QueryLimits 

type QueryLimits struct {
	MaxWireBytes  int
	MaxDepth      int
	MaxNodes      int
	MaxArrayItems int
	MaxValueBytes int
	MaxSortFields int
	MaxLimit      int
}

type QueryOptions 

type QueryOptions struct {
	Sort  []SortField
	Skip  int
	Limit *int
}

type QuerySnapshot 

type QuerySnapshot struct {
	Token     uint64
	Documents []Document
}

type QuerySpec 

type QuerySpec struct {
	// contains filtered or unexported fields
}

func CompileQuery 

func CompileQuery(filter Filter, options QueryOptions) (QuerySpec, error)

func DecodeQuerySpecJSON 

func DecodeQuerySpecJSON(data []byte, limits QueryLimits) (QuerySpec, error)

func (QuerySpec) Capped 

func (q QuerySpec) Capped(max int) QuerySpec

func (QuerySpec) Constrain 

func (q QuerySpec) Constrain(policy QuerySpec) QuerySpec

Constrain applies a server-owned row predicate before the caller's sort and pagination. This is the safe composition point for authorization policies.

func (QuerySpec) Execute 

func (q QuerySpec) Execute(documents []Document) []Document

func (QuerySpec) HasModifiers 

func (q QuerySpec) HasModifiers() bool

func (QuerySpec) Limit 

func (q QuerySpec) Limit() (int, bool)

func (QuerySpec) Match 

func (q QuerySpec) Match(document Document) bool

func (QuerySpec) Paths 

func (q QuerySpec) Paths() []string

func (QuerySpec) Skip 

func (q QuerySpec) Skip() int

func (QuerySpec) Sort 

func (q QuerySpec) Sort() []SortField

type QuerySubscription 

type QuerySubscription struct {
	Snapshots <-chan QuerySnapshot
	Errors    <-chan error
	// contains filtered or unexported fields
}

func (*QuerySubscription) Close 

func (s *QuerySubscription) Close()

type SortField 

type SortField struct {
	Path      string `json:"path"`
	Direction int    `json:"direction"`
}

type Update 

type Update map[string]any

type UpdateResult 

type UpdateResult struct{ MatchedCount, ModifiedCount int64 }

type Value 

type Value struct {
	// contains filtered or unexported fields
}

Value is a closed tagged value. Its representation is private so callers cannot construct a tag/payload mismatch.

func Array 

func Array(v ...Value) Value

func Binary 

func Binary(v []byte) Value

func Bool 

func Bool(v bool) Value

func Float 

func Float(v float64) Value

func ID 

func ID(v DocumentID) Value

func Int 

func Int(v int64) Value

func Null 

func Null() Value

func Object 

func Object(v Document) Value

func String 

func String(v string) Value

func Time 

func Time(v time.Time) Value

Time stores millisecond precision, matching JavaScript Date and the wire contract. Precision is normalized at construction rather than silently lost during transport.

func ValueOf 

func ValueOf(x any) (Value, error)

func (Value) ArrayValue 

func (v Value) ArrayValue() ([]Value, bool)

func (Value) BinaryValue 

func (v Value) BinaryValue() ([]byte, bool)

func (Value) Bool 

func (v Value) Bool() (bool, bool)

func (Value) Clone 

func (v Value) Clone() Value

func (Value) Equal 

func (v Value) Equal(other Value) bool

func (Value) Float64 

func (v Value) Float64() (float64, bool)

func (Value) IDValue 

func (v Value) IDValue() (DocumentID, bool)

func (Value) Int64 

func (v Value) Int64() (int64, bool)

func (Value) Kind 

func (v Value) Kind() Kind

func (Value) ObjectValue 

func (v Value) ObjectValue() (Document, bool)

func (Value) StringValue 

func (v Value) StringValue() (string, bool)

func (Value) TimeValue 

func (v Value) TimeValue() (time.Time, bool)

Source Files

View all Source files

Directories

Path Synopsis
cmd
meld command
internal
index
server
storage
wal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.