latticedb

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 30, 2026 License: MIT Imports: 9 Imported by: 0

README

LatticeDB Go

An embedded graph database written entirely in Go. It provides transactional graph operations, Cypher-style queries, full-text search, vector search, durable WAL recovery, streams, exports, and online snapshots without cgo.

Install

LatticeDB Go requires Go 1.27 or newer.

go get github.com/mrchypark/latticedb-go@v0.1.0

Quick start

package main

import (
	"fmt"
	"log"

	latticedb "github.com/mrchypark/latticedb-go"
)

func main() {
	db, err := latticedb.Open("app.ltdb", latticedb.OpenOptions{Create: true})
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	err = db.Update(func(tx *latticedb.Tx) error {
		_, err := tx.CreateNode(latticedb.CreateNodeOptions{
			Labels:     []string{"Person"},
			Properties: map[string]latticedb.Value{"name": "Ada"},
		})
		return err
	})
	if err != nil {
		log.Fatal(err)
	}

	result, err := db.Query("MATCH (n:Person) RETURN n.name", nil)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(result.Rows)
}

Highlights

  • Pure Go on Linux, macOS, and Windows
  • ACID write transactions with WAL recovery and checkpoints
  • Transaction-scoped queries and binary-safe application metadata
  • Property indexes, full-text search, and exact or HNSW vector search
  • Online frozen-generation backups through BeginSnapshot
  • JSON, JSONL, CSV, and DOT export
  • Context cancellation and row, work, and logical-byte budgets

Storage and transaction contract

  • v0.1 uses the new state v4 and WAL v3 formats. Older metadata-free state v3 and WAL v2 files are readable, but new files intentionally fail closed in older binaries.
  • A Tx is single-owner and must not be used concurrently.
  • Commit and CommitContext are one-shot: the transaction becomes inactive whether the commit succeeds or fails.
  • Only one online snapshot may be active per database. Writers can continue after the snapshot generation is captured.

The detailed behavioral contract is documented in docs/engine_conformance.md, with the value model in docs/value_model.md.

Development

go test ./...
go test -race ./...
(cd conformance/go && go test ./...)

License

MIT

Documentation

Overview

Package latticedb will host the pure Go LatticeDB engine.

The repo is being bootstrapped contract-first. The engine conformance spec and extracted conformance suite are in docs/ and conformance/go/; the concrete engine API and storage implementation will be built against that contract.

Index

Constants

View Source 
const (
	// Deprecated: use embedding.APIFormatOllama.
	EmbeddingAPIFormatOllama = latticeembedding.APIFormatOllama
	// Deprecated: use embedding.APIFormatOpenAI.
	EmbeddingAPIFormatOpenAI = latticeembedding.APIFormatOpenAI
)

Variables

View Source 
var (
	// These aliases preserve the upstream sentinel names. Pure-Go errors may
	// still carry richer engine-specific sentinels where appropriate.
	ErrReadOnlyDatabase = ErrReadOnly
	ErrReadOnlyTx       = ErrReadOnly
)
View Source 
var (
	ErrReadOnly                       = engine.ErrReadOnly
	ErrWriteTxActive                  = engine.ErrWriteTxActive
	ErrManagedTransaction             = engine.ErrManagedTransaction
	ErrInactiveTx                     = engine.ErrInactiveTx
	ErrDatabaseLocked                 = engine.ErrDatabaseLocked
	ErrDatabaseLayoutConflict         = engine.ErrDatabaseLayoutConflict
	ErrDatabaseClosed                 = engine.ErrDatabaseClosed
	ErrTransactionsActive             = engine.ErrTransactionsActive
	ErrSnapshotActive                 = engine.ErrSnapshotActive
	ErrWriteConflict                  = engine.ErrWriteConflict
	ErrRecoveryRequired               = engine.ErrRecoveryRequired
	ErrResourceLimit                  = engine.ErrResourceLimit
	ErrAlreadyExists                  = engine.ErrAlreadyExists
	ErrInvalidArgument                = engine.ErrInvalidArgument
	ErrVectorIndexMaintenanceRequired = engine.ErrVectorIndexMaintenanceRequired
	ErrUnsupportedOption              = engine.ErrUnsupportedOption
	ErrCommitOutcomeUnknown           = engine.ErrCommitOutcomeUnknown
)
View Source 
var ErrEmbeddingClosed = errors.New("embedding client is closed")

ErrEmbeddingClosed is returned when Embed is called after Close.

Deprecated: use embedding.ErrClosed.

Functions

func Dump 

func Dump(dbPath string) ([]byte, error)

func DumpContext 

func DumpContext(ctx context.Context, dbPath string) ([]byte, error)

func Export 

func Export(dbPath string, format ExportFormat, outputPath string) ([]byte, error)

func ExportContext 

func ExportContext(ctx context.Context, dbPath string, format ExportFormat, outputPath string) ([]byte, error)

func HashEmbed deprecated 

func HashEmbed(text string, dimensions uint16) ([]float32, error)

HashEmbed returns the deterministic built-in hash embedding.

Deprecated: use embedding.Hash.

func SimulateCrash 

func SimulateCrash(dbPath string) error

func Version 

func Version() string

Version returns the Go module version embedded in the calling binary.

Types

type CreateEdgeOptions 

type CreateEdgeOptions struct {
	Properties map[string]Value
}

type CreateNodeOptions 

type CreateNodeOptions struct {
	Labels     []string
	Properties map[string]Value
}

type DB 

type DB struct {
	// contains filtered or unexported fields
}

func Deserialize 

func Deserialize(data []byte, opts OpenOptions) (*DB, error)

Deserialize opens a database from bytes returned by Serialize.

func Open 

func Open(path string, opts OpenOptions) (*DB, error)

Open opens a directory-backed database or a regular database file previously returned by Serialize.

func OpenContext 

func OpenContext(ctx context.Context, path string, opts OpenOptions) (*DB, error)

func (*DB) Begin 

func (db *DB) Begin(readOnly bool) (*Tx, error)

func (*DB) BeginRead 

func (db *DB) BeginRead() (*Tx, error)

func (*DB) BeginSnapshot 

func (db *DB) BeginSnapshot() (*Snapshot, error)

BeginSnapshot pins one committed generation while database writes continue. Only one Snapshot may be active for a DB.

func (*DB) BeginWrite 

func (db *DB) BeginWrite() (*Tx, error)

func (*DB) CacheClear 

func (db *DB) CacheClear() error

func (*DB) CacheStats 

func (db *DB) CacheStats() (QueryCacheStats, error)

func (*DB) Changes 

func (db *DB) Changes(afterSequence uint64, limit uint, timeoutMS uint32) ([]StreamRecord, error)

func (*DB) Checkpoint 

func (db *DB) Checkpoint() error

func (*DB) Close 

func (db *DB) Close() error

func (*DB) CreateEdgePropertyIndex 

func (db *DB) CreateEdgePropertyIndex(edgeType, property string) error

func (*DB) CreateNodePropertyIndex 

func (db *DB) CreateNodePropertyIndex(label, property string) error

func (*DB) DropEdgePropertyIndex 

func (db *DB) DropEdgePropertyIndex(edgeType, property string) error

func (*DB) DropNodePropertyIndex 

func (db *DB) DropNodePropertyIndex(label, property string) error

func (*DB) Dump 

func (db *DB) Dump() ([]byte, error)

func (*DB) DumpContext 

func (db *DB) DumpContext(ctx context.Context) ([]byte, error)

func (*DB) DumpTo 

func (db *DB) DumpTo(output io.Writer) error

func (*DB) DumpToContext 

func (db *DB) DumpToContext(ctx context.Context, output io.Writer) error

DumpToContext observes cancellation between writes. It cannot interrupt an output writer that is itself blocked in Write.

func (*DB) Export 

func (db *DB) Export(format ExportFormat, outputPath string) ([]byte, error)

func (*DB) ExportContext 

func (db *DB) ExportContext(ctx context.Context, format ExportFormat, outputPath string) ([]byte, error)

func (*DB) ExportTo 

func (db *DB) ExportTo(format ExportFormat, output io.Writer) error

func (*DB) ExportToContext 

func (db *DB) ExportToContext(ctx context.Context, format ExportFormat, output io.Writer) error

ExportToContext observes cancellation between writes. It cannot interrupt an output writer that is itself blocked in Write.

func (*DB) FTSSearch 

func (db *DB) FTSSearch(query string, opts FTSSearchOptions) ([]FTSSearchResult, error)

func (*DB) FTSSearchContext 

func (db *DB) FTSSearchContext(ctx context.Context, query string, opts FTSSearchOptions) ([]FTSSearchResult, error)

func (*DB) FTSSearchFuzzy 

func (db *DB) FTSSearchFuzzy(query string, opts FTSSearchOptions) ([]FTSSearchResult, error)

func (*DB) GetNodesByLabel 

func (db *DB) GetNodesByLabel(label string) ([]NodeID, error)

func (*DB) GetStreamOffset 

func (db *DB) GetStreamOffset(stream, consumer string) (uint64, bool, error)

func (*DB) IsOpen 

func (db *DB) IsOpen() bool

func (*DB) Path 

func (db *DB) Path() string

func (*DB) Query 

func (db *DB) Query(query string, params map[string]Value) (QueryResult, error)

func (*DB) QueryContext 

func (db *DB) QueryContext(ctx context.Context, query string, params map[string]Value, opts QueryOptions) (QueryResult, error)

func (*DB) ReadStream 

func (db *DB) ReadStream(stream string, afterSequence uint64, limit uint, timeoutMS uint32) ([]StreamRecord, error)

func (*DB) RebuildVectorIndexContext 

func (db *DB) RebuildVectorIndexContext(ctx context.Context) error

func (*DB) Serialize 

func (db *DB) Serialize() ([]byte, error)

Serialize returns a standalone database file. Writing the bytes to a regular file produces a path that Open can read and update.

func (*DB) Update 

func (db *DB) Update(fn func(*Tx) error) error

func (*DB) UpdateContext 

func (db *DB) UpdateContext(ctx context.Context, fn func(*Tx) error) error

func (*DB) VectorIndexStats 

func (db *DB) VectorIndexStats() (VectorIndexStats, error)

func (*DB) VectorSearch 

func (db *DB) VectorSearch(vector []float32, opts VectorSearchOptions) ([]VectorSearchResult, error)

func (*DB) VectorSearchContext 

func (db *DB) VectorSearchContext(ctx context.Context, vector []float32, opts VectorSearchOptions) ([]VectorSearchResult, error)

func (*DB) View 

func (db *DB) View(fn func(*Tx) error) error

type DurabilityMode 

type DurabilityMode uint8
const (
	DurabilityStandard DurabilityMode = iota
	DurabilityFull
)

type Edge 

type Edge struct {
	ID         uint64
	SourceID   uint64
	TargetID   uint64
	Type       string
	Properties map[string]Value
}

type EdgeID 

type EdgeID = uint64

type EmbeddingAPIFormat deprecated 

type EmbeddingAPIFormat = latticeembedding.APIFormat

EmbeddingAPIFormat selects the wire format used by an embedding endpoint.

Deprecated: use package github.com/mrchypark/latticedb-go/embedding.

type EmbeddingClient deprecated 

type EmbeddingClient struct {
	// contains filtered or unexported fields
}

EmbeddingClient is an optional HTTP embedding client.

Deprecated: use embedding.Client.

func NewEmbeddingClient deprecated 

func NewEmbeddingClient(config EmbeddingConfig) (*EmbeddingClient, error)

NewEmbeddingClient creates an HTTP embedding client.

Deprecated: use embedding.NewClient.

func (*EmbeddingClient) Close 

func (client *EmbeddingClient) Close() error

Close releases this client. It is safe to call more than once.

func (*EmbeddingClient) Embed 

func (client *EmbeddingClient) Embed(text string) ([]float32, error)

Embed requests one vector from the configured endpoint.

type EmbeddingConfig deprecated 

type EmbeddingConfig = latticeembedding.Config

EmbeddingConfig configures an optional HTTP embedding client.

Deprecated: use embedding.Config.

type Error 

type Error struct {
	Code    ErrorCode
	Message string
	// contains filtered or unexported fields
}

Error is a structured database error.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Is 

func (e *Error) Is(target error) bool

func (*Error) Unwrap 

func (e *Error) Unwrap() error

type ErrorCode 

type ErrorCode int

ErrorCode identifies a low-level database error. Values match the upstream Go binding so callers can keep using errors.As and code comparisons. 

const (
	ErrorOK              ErrorCode = 0
	ErrorGeneric         ErrorCode = -1
	ErrorIO              ErrorCode = -2
	ErrorCorruption      ErrorCode = -3
	ErrorNotFound        ErrorCode = -4
	ErrorAlreadyExists   ErrorCode = -5
	ErrorInvalidArg      ErrorCode = -6
	ErrorTxnAborted      ErrorCode = -7
	ErrorLockTimeout     ErrorCode = -8
	ErrorReadOnly        ErrorCode = -9
	ErrorFull            ErrorCode = -10
	ErrorVersionMismatch ErrorCode = -11
	ErrorChecksum        ErrorCode = -12
	ErrorOutOfMemory     ErrorCode = -13
	ErrorUnsupported     ErrorCode = -14
	ErrorValueTooLarge   ErrorCode = -15
	ErrorDatabaseLocked  ErrorCode = -16
)

type ExportFormat 

type ExportFormat string
const (
	ExportFormatJSON  ExportFormat = "json"
	ExportFormatJSONL ExportFormat = "jsonl"
	ExportFormatCSV   ExportFormat = "csv"
	ExportFormatDOT   ExportFormat = "dot"
)

type FTSSearchOptions 

type FTSSearchOptions struct {
	Limit         uint32
	MaxDistance   uint32
	MinTermLength uint32
	MaxWork       uint64
	MaxBytes      uint64
}

type FTSSearchResult 

type FTSSearchResult = engine.FTSSearchResult

type Node 

type Node struct {
	ID         uint64
	Labels     []string
	Properties map[string]Value
}

type NodeID 

type NodeID = uint64

type OpenOptions 

type OpenOptions struct {
	Create                      bool
	ReadOnly                    bool
	CacheSizeMB                 uint32
	PageSize                    uint32
	EnableWAL                   bool
	DisableWAL                  bool
	EnableAdjacencyCache        bool
	EnableVectors               bool
	EnableVector                bool
	DisableLock                 bool
	VectorIndexMode             VectorIndexMode
	VectorDimensions            uint16
	Durability                  DurabilityMode
	WALCheckpointThresholdBytes uint64
	// MaxDatabaseSnapshotBytes is a conservative upper bound for the canonical streamed snapshot payload.
	MaxDatabaseSnapshotBytes uint64
	// VectorIndexBuildMaxWork bounds synchronous HNSW distance work.
	VectorIndexBuildMaxWork uint64
	// VectorIndexBuildMaxLogicalBytes bounds conservative current+new index metadata, not process RSS.
	VectorIndexBuildMaxLogicalBytes uint64
	// DerivedIndexBuildMaxWork bounds label, edge, adjacency, and FTS index rebuild work during Open.
	DerivedIndexBuildMaxWork uint64
	// DerivedIndexBuildMaxLogicalBytes bounds conservative derived posting metadata, not process RSS.
	DerivedIndexBuildMaxLogicalBytes uint64
}

type QueryCacheStats 

type QueryCacheStats struct {
	Entries uint32
	Hits    uint64
	Misses  uint64
}

type QueryError 

type QueryError struct {
	Code           ErrorCode
	Stage          QueryErrorStage
	Message        string
	DiagnosticCode string
	Location       *QueryErrorLocation
	// contains filtered or unexported fields
}

QueryError is a structured query parsing, planning, or execution error.

func (*QueryError) Error 

func (e *QueryError) Error() string

func (*QueryError) Unwrap 

func (e *QueryError) Unwrap() error

type QueryErrorLocation 

type QueryErrorLocation struct {
	Line   uint32
	Column uint32
	Length uint32
}

QueryErrorLocation identifies the source span associated with a query error.

type QueryErrorStage 

type QueryErrorStage int

QueryErrorStage identifies the phase in which a query failed. 

const (
	QueryErrorStageNone      QueryErrorStage = 0
	QueryErrorStageParse     QueryErrorStage = 1
	QueryErrorStageSemantic  QueryErrorStage = 2
	QueryErrorStagePlan      QueryErrorStage = 3
	QueryErrorStageExecution QueryErrorStage = 4
)

type QueryOptions 

type QueryOptions struct {
	MaxRows uint64
	MaxWork uint64
	// MaxBytes limits logical query materialization, not the process RSS.
	MaxBytes uint64
}

type QueryResult 

type QueryResult struct {
	Columns []string
	Rows    []map[string]Value
}

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is one fixed committed database generation.

func (*Snapshot) Backup 

func (snapshot *Snapshot) Backup(path string) error

Backup writes the frozen generation as a standalone regular database file.

func (*Snapshot) Close 

func (snapshot *Snapshot) Close() error

Close releases the frozen generation. Close is idempotent.

type StreamRecord 

type StreamRecord = engine.StreamRecord

type Tx 

type Tx struct {
	// contains filtered or unexported fields
}

Tx is a single-owner transaction handle. Its methods must not be called concurrently; use separate read transactions for concurrent work.

func (*Tx) BatchInsert deprecated 

func (tx *Tx) BatchInsert(label string, vectors [][]float32) ([]uint64, error)

Deprecated: use BatchInsertVectors. Earliest removal is v0.6.0.

func (*Tx) BatchInsertVectors 

func (tx *Tx) BatchInsertVectors(label string, vectors [][]float32) ([]uint64, error)

BatchInsertVectors inserts multiple vector-bearing nodes in a single call.

func (*Tx) Commit 

func (tx *Tx) Commit() error

Commit makes the transaction inactive, whether it succeeds or fails.

func (*Tx) CommitContext 

func (tx *Tx) CommitContext(ctx context.Context) error

CommitContext makes the transaction inactive, whether it succeeds or fails.

func (*Tx) CreateEdge 

func (tx *Tx) CreateEdge(sourceID uint64, targetID uint64, edgeType string, opts CreateEdgeOptions) (Edge, error)

func (*Tx) CreateNode 

func (tx *Tx) CreateNode(opts CreateNodeOptions) (Node, error)

func (*Tx) DeleteAppMetadata 

func (tx *Tx) DeleteAppMetadata(key []byte) error

func (*Tx) DeleteEdge 

func (tx *Tx) DeleteEdge(sourceID, targetID NodeID, edgeType string) error

func (*Tx) DeleteNode 

func (tx *Tx) DeleteNode(nodeID uint64) error

func (*Tx) FTSIndex 

func (tx *Tx) FTSIndex(nodeID uint64, text string) error

func (*Tx) FTSIndexContext 

func (tx *Tx) FTSIndexContext(ctx context.Context, nodeID uint64, text string) error

func (*Tx) FindEdgesByTypeProperty 

func (tx *Tx) FindEdgesByTypeProperty(edgeType, property string, value Value, limit uint) ([]uint64, error)

func (*Tx) FindNodesByLabelProperty 

func (tx *Tx) FindNodesByLabelProperty(label, property string, value Value, limit uint) ([]uint64, error)

func (*Tx) GetAppMetadata 

func (tx *Tx) GetAppMetadata(key []byte) ([]byte, bool, error)

func (*Tx) GetEdgeProperty 

func (tx *Tx) GetEdgeProperty(edgeID uint64, key string) (Value, bool, error)

func (*Tx) GetIncomingEdges 

func (tx *Tx) GetIncomingEdges(nodeID uint64) ([]Edge, error)

func (*Tx) GetIncomingEdgesByType 

func (tx *Tx) GetIncomingEdgesByType(nodeID uint64, edgeType string, limit uint) ([]Edge, error)

func (*Tx) GetNode 

func (tx *Tx) GetNode(nodeID uint64) (*Node, error)

func (*Tx) GetOutgoingEdges 

func (tx *Tx) GetOutgoingEdges(nodeID uint64) ([]Edge, error)

func (*Tx) GetOutgoingEdgesByType 

func (tx *Tx) GetOutgoingEdgesByType(nodeID uint64, edgeType string, limit uint) ([]Edge, error)

func (*Tx) GetProperty 

func (tx *Tx) GetProperty(nodeID uint64, key string) (Value, bool, error)

func (*Tx) IsActive 

func (tx *Tx) IsActive() bool

func (*Tx) IsReadOnly 

func (tx *Tx) IsReadOnly() bool

func (*Tx) NodeExists 

func (tx *Tx) NodeExists(nodeID uint64) (bool, error)

func (*Tx) PublishStream 

func (tx *Tx) PublishStream(stream, kind string, payload Value) error

func (*Tx) PublishStreamGetSequence 

func (tx *Tx) PublishStreamGetSequence(stream, kind string, payload Value) (uint64, error)

func (*Tx) PutAppMetadata 

func (tx *Tx) PutAppMetadata(key, value []byte) error

func (*Tx) Query 

func (tx *Tx) Query(query string, params map[string]Value) (QueryResult, error)

func (*Tx) QueryContext 

func (tx *Tx) QueryContext(ctx context.Context, query string, params map[string]Value, opts QueryOptions) (QueryResult, error)

func (*Tx) RemoveEdgeProperty 

func (tx *Tx) RemoveEdgeProperty(edgeID uint64, key string) error

func (*Tx) Rollback 

func (tx *Tx) Rollback() error

func (*Tx) SetEdgeProperty 

func (tx *Tx) SetEdgeProperty(edgeID uint64, key string, value Value) error

func (*Tx) SetProperty 

func (tx *Tx) SetProperty(nodeID uint64, key string, value Value) error

func (*Tx) SetStreamOffset 

func (tx *Tx) SetStreamOffset(stream, consumer string, sequence uint64) error

func (*Tx) SetVector 

func (tx *Tx) SetVector(nodeID uint64, key string, vector []float32) error

func (*Tx) TrimStream 

func (tx *Tx) TrimStream(stream string, beforeSequence uint64) error

type Value 

type Value = any

type VectorIndexMode 

type VectorIndexMode uint8
const (
	// VectorIndexExactOnly is the safe zero-value: no derived index build or memory overhead.
	VectorIndexExactOnly VectorIndexMode = iota
	// VectorIndexHNSWSynchronous builds the approximate index before Open returns.
	VectorIndexHNSWSynchronous
)

type VectorIndexStats 

type VectorIndexStats struct {
	LiveEntries                uint64
	IndexEntries               uint64
	Tombstones                 uint64
	TombstoneBytes             uint64
	TombstoneBytesUntilRebuild uint64
	MutationDebt               uint64
	RebuildThreshold           uint64
	DebtUntilRebuild           uint64
	EstimatedBuildLogicalBytes uint64
	ExactFallbacks             uint64
	Rebuilds                   uint64
	RebuildNanoseconds         uint64
}

type VectorSearchOptions 

type VectorSearchOptions struct {
	K        uint32
	EfSearch uint16
	// Exact disables the approximate index and scans every vector.
	Exact bool
	// MaxWork and MaxBytes bound one direct search request's scalar work and logical scratch/result bytes, not process RSS. Zero MaxWork means no caller-requested work limit; zero MaxBytes uses a 64 MiB logical scratch limit.
	MaxWork  uint64
	MaxBytes uint64
}

type VectorSearchResult 

type VectorSearchResult = engine.VectorSearchResult

Source Files

View all Source files

Directories

Path Synopsis
Package embedding provides optional deterministic and HTTP embedding helpers.
 Package embedding provides optional deterministic and HTTP embedding helpers.
internal
engine
exporter
search
store

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.