yuruidb

README

yuruidb

English | 日本語

A Go 1.27+ record storage library with interchangeable, statically imported backends. One Store owns a connection; generic methods select the registered data type and collection. JSON data uses Record[T] and Page[T]. IDs may be validated names or generated UUIDv7 strings. Listing uses ascending ID byte order.

Cloud drivers use standard-library HTTP, without cloud SDK dependencies. The core has no external imports. bbolt is an optional driver dependency; pure-Go SQL test dependencies live in the separate integration module.

The module path is github.com/shibukawa/yuruidb.

Usage

import (
    "context"
    "github.com/shibukawa/yuruidb"
    _ "github.com/shibukawa/yuruidb/drivers/bbolt"
)

type Job struct { Text string `json:"text"` }
type Waiting Job
type Done Job

ctx := context.Background()
store, err := yuruidb.Open(ctx, "bbolt", yuruidb.Config{"Path": "spool.db"})
// Handle errors in application code.
defer store.Close()
err = store.Bind[Waiting]("jobs")
err = store.Bind[Done]("done")

id, err := yuruidb.NewID()
record, err := store.Insert(ctx, id, Waiting{Text: "hello"})
record, err = store.Get[Waiting](ctx, id)
record, err = store.Update(ctx, id, Waiting{Text: "updated"}, record.Revision())
page, err := store.List[Waiting](ctx, yuruidb.ListOptions{Limit: 100})
err = store.Delete[Waiting](ctx, id, record.Revision())
// record.ID(), record.Revision(), record.Data.Text

See examples/basic for runnable error handling. Drivers also expose Open(ctx, Config) through named imports. Bind validates registrations without I/O; the first operation creates its internal collection view. Unregistered types return ErrInvalid. Rebinding a type to another name or binding a different type to the same name is rejected. Pointer and value forms share a binding. type Done Job defines a distinct type; type Done = Job is an alias and shares Job's identity. Defined types retain fields/tags, but do not inherit Job's own methods.

Data and concurrency

T can be any JSON-compatible struct, pointer, map, slice, scalar, json.RawMessage, or custom JSON type. Writes marshal and unmarshal before mutation, returning ErrInvalid for unsupported values. Use compatible schemas across clients. Returned data does not alias ordinary caller-owned maps/slices. Metadata is read-only in Go; record JSON retains id, revision, and data.

• IDs contain 1–256 ASCII letters, digits, ., _, or -; ., .., and Firestore-reserved __…__ names are rejected. Case is preserved. NewID() remains a process-monotonic UUIDv7 generator. UUIDv7 order is not exact cross-process commit order; names sort by bytes (10 before 2). Updates preserve IDs. Pages have no snapshot guarantee.
• Insert never overwrites a live ID. It attempts conditional creation first, reading only on conflict. Update and Delete require a retrieved opaque revision. Revisions bind collection, ID, random write nonce, and native CAS token; current tokens avoid a pre-write GET. Backends that cannot return native tokens use readback when needed. Conflicts never adopt a newer revision automatically.
• Delete uses conditional physical deletion on capable backends. Others retain conditional tombstones. An ID may be recreated; earlier record revisions remain stale. Existing tombstones can be removed with store.PurgeTombstones[T](ctx) on capable backends. Bucket version history and bbolt file compaction are separate storage concerns.
• Value-bearing listing avoids per-item reads on bbolt, memory, SQL/D1, DynamoDB, and Firestore. S3/R2/GCS enumerate keys and read records; Firebase needs individual ETags. Listing may observe concurrent changes.
• bbolt shares a DB per normalized file path, including symlink resolution, with reference-counted close. Five types can use five buckets in one file. Values use a compact binary envelope without base64 wrapping. It remains a local file backend, not a cross-process remote store.

Unknown outcomes

A conditional mutation may commit even if its response is lost. The record layer attempts a readback when context and consistency permit. If it finds the attempted nonce, it returns success. Otherwise, errors.Is(err, yuruidb.ErrOutcomeUnknown) remains true; ErrConflict and ErrAlreadyExists do not match, even when a replay encountered a failed condition. Do not reapply the mutation before reconciling its outcome. errors.As exposes *yuruidb.MutationError with the operation, collection, ID, nonce, expected revision, and attempted revision.

var attempt *yuruidb.MutationError
if errors.As(err, &attempt) {
    current, readErr := store.Get[Waiting](recoveryCtx, attempt.ID)
    if readErr == nil && attempt.MatchesRevision(current.Revision()) {
        // This exact attempted write is currently stored.
    }
}

A mismatch does not prove failure: another writer may already have superseded it. On eventually consistent KV, a matching read confirms visibility of that version, not global current state. Absence after physical deletion confirms the desired state, not which deletion committed. If the original context expired, reconcile with an explicit new context.

Backends and authentication

Driver	Main connection settings	Conditional record mutations
memory	none	in-process
bbolt	Path, Timeout, InitialMmapSize	local file transactions
sql	DB, Dialect, AutoMigrate, CloseDB	SQLite/PostgreSQL SQL dialects
s3	Bucket, Region, optional Endpoint, AddressStyle	ETag conditions
dynamodb	Table, Region	conditional PutItem/DeleteItem
gcs	Bucket, Token	generation conditions
firestore	Project, Database, Token	updateTime conditions
firebase	Endpoint, Token	ETag conditions, including deletion
r2	Endpoint, Bucket, Credentials	verified S3 conditions
d1	AccountID, DatabaseID, Token, AutoMigrate	conditional SQL
cfkv	AccountID, NamespaceID, Token	unsupported; explicit unconditional Set/Remove only

Connection configs omit Collection for the typed Store; OpenRaw can still select one collection. HTTP clients and authorizers are injectable. AWS's default credential provider checks environment credentials, web identity/STS (EKS IRSA), container credentials (ECS/EKS Pod Identity), then EC2 IMDSv2, caching and refreshing temporary credentials. Profiles and SSO remain caller-injected. Region order is explicit config, AWS_REGION, AWS_DEFAULT_REGION, then us-east-1; R2 defaults to auto.

AWS S3 uses virtual hosting by default for suitable bucket names; dotted names use path style for TLS compatibility. AddressStyle can be auto, virtual, or path. Custom endpoints default to path style. Opening a custom S3/R2 endpoint creates temporary probe keys under .yuruidb-probes/ outside every valid collection, verifies conditional creation/replacement, concurrent CAS, and conditional deletion, then cleans up. Cleanup uses a cancellation-independent context with a 10-second timeout. Cleanup failures fail Open, and any leftover probe cannot enter a record listing. IAM must allow these probe operations under the reserved prefix. Ignored PUT conditions fail Open; unsupported/ignored DELETE conditions select tombstones. Set VerifyConditions: true to probe AWS endpoints as well. s3.Verify/r2.Verify expose a diagnostic report. Probes are bounded observations, not a proof of every service guarantee.

Transient HTTP 429/500/502/503/504 responses receive bounded backoff with jitter and fresh authorization. DynamoDB also retries HTTP 400 ThrottlingException, ProvisionedThroughputExceededException, and RequestLimitExceeded responses (AWS error reference). Only conditional mutations are replayed, with unchanged body and condition. Unconditional KV writes and D1 mutations are not blindly retried. Retry-After is respected; long minimum delays return the error. JSON responses are limited to 32 MiB; provider limits and encoding overhead can be smaller. Oversized S3/GCS record writes are rejected before upload.

Streaming documents

sql.OpenBlobs, bbolt.OpenBlobs, s3.OpenBlobs, r2.OpenBlobs, and gcs.OpenBlobs open an explicitly named collection for immutable documents. Insert(ctx, id, io.Reader) creates an absent ID; HTTP uploads spool to a temporary file for bounded-memory signing/replay. bbolt stores 64 KiB chunks in reserved .yuruidb-blobs/<collection> buckets in the shared file, isolated from record buckets. bbolt reads input outside transactions and commits staging data in batches of at most 2 MiB. A final short transaction moves the completed bucket into the collection atomically; partial uploads are invisible. Failed or canceled uploads are cleaned up, and the first owner opening a file removes staging left by interrupted uploads. Additional handles never sweep another live upload. The JSON response limit does not apply.

Open(ctx, id) returns a streaming reader that also implements io.ReaderAt; keep its context alive and close it when finished. HTTP Open retrieves metadata only (HEAD on S3/R2, the JSON object resource on GCS). The sequential GET starts on the first nonempty Read, guarded by the observed version; ReadAt-only callers never open a full-document stream. Close cancels pending reader requests. HTTP ReadAt sends a range request guarded by the observed ETag/generation; an endpoint ignoring ranges returns ErrUnsupported. bbolt range reads use short read transactions. List(ctx, yuruidb.ListOptions{Limit: 100}) returns blob.Page{Blobs: []blob.Metadata, Cursor: ...} in ascending ID order, including version and size, without downloading bodies. Use the cursor for the next page and compare IDs with record references to find possible orphans. Pages have no snapshot guarantee; coordinate cleanup with producers so pending uploads are not deleted.

Store the returned ID/version reference in a small typed record. Use fresh IDs; native blob ETags can repeat for identical bytes. Conditional Delete(ctx, id, revision) requires backend support. Explicit Remove(ctx, id) performs an unconditional deletion even without conditional DELETE support; only use it when IDs are never reused. Document and record operations do not form a transaction.

blobs, err := s3.OpenBlobs(ctx, s3.Config{Bucket: "spool", Collection: "documents"})
meta, err := blobs.Insert(ctx, documentID, sourceReader)
reader, err := blobs.Open(ctx, meta.ID)
defer reader.Close()
buf := make([]byte, 4096)
n, err := reader.ReadAt(buf, 1024)
page, err := blobs.List(ctx, yuruidb.ListOptions{Limit: 100})
// Conditional deletion: blobs.Delete(ctx, meta.ID, meta.Revision)
// With never-reused IDs: blobs.Remove(ctx, meta.ID)

SQL Blob stores use the existing SQL driver's Config and return the same *blob.Store. The default table is ururidb_blobs, separate from ururidb_records; custom Table names must also remain separate. With AutoMigrate: true, SQLite creates a BLOB column and PostgreSQL a BYTEA column. Bodies are stored as binary, without JSON/base64 wrapping. Open and List read metadata only; Read and ReadAt fetch byte ranges with SQL substr (SQLite, PostgreSQL). Each range checks the random revision, so reads detect ID replacement. SQL inserts buffer the entire input in memory and use one atomic insert; database size limits still apply.

// sqldriver is github.com/shibukawa/yuruidb/drivers/sql.
blobs, err := sqldriver.OpenBlobs(ctx, sqldriver.Config{
    DB: db, Dialect: "postgres", Collection: "documents", AutoMigrate: true,
    Identity: "my-app-primary",
})
// Insert/Open/ReadAt/List/Delete/Remove are identical across Blob backends.

An injected SQL DB is borrowed by default. When sharing it between record and Blob stores, keep CloseDB: false and close the DB after both stores. Assign distinct stable Identity values to different databases to keep their cursors separate. Record and Blob operations remain separate transactions.

Live bbolt backup

store.Backup(ctx, writer) exports a consistent snapshot of the whole bbolt file while it remains open, including every record collection and Blob bucket. The caller closes the writer and handles its errors. The Store lock is released before output, so a slow writer does not block record lookups or binding. Close waits for the active bbolt transaction. Write to a separate file; a long backup retains a read transaction and can delay file growth. Cancellation is checked between writes and cannot interrupt a blocked caller-provided writer. Other backends return ErrUnsupported. Restore by opening the completed snapshot at a new path; use bbolt tools on that snapshot for compaction. Record revisions are scoped to the file path; retrieve them again after restoring at a different path.

// store was opened with the bbolt driver; destination is a separate io.Writer.
err = store.Backup(ctx, destination)

bbolt.Config.InitialMmapSize sets the initial mmap capacity in bytes for both record and Blob connections. It defaults to 0, preserving bbolt's behavior. An explicit capacity large enough for subsequent writes can avoid remapping waits during a backup; growth beyond the mapping can still wait. On Windows, bbolt immediately extends the database file to the requested mapping size, so a large capacity is opt-in. All handles sharing one file must use the same Mode, Timeout, and InitialMmapSize; mismatches return ErrInvalid. Negative capacities are rejected.

store, err := bbolt.Open(ctx, bbolt.Config{
    Path: "spool.db", InitialMmapSize: 256 << 20, // explicit 256 MiB mapping
})
// For build-tag-selected drivers, the same field is available via:
// yuruidb.Config{"Path": "spool.db", "InitialMmapSize": 256 << 20}

For an HTTP backup endpoint with slow clients, write the snapshot to a local temporary file, finish Backup, seek to the start, and then stream that file to the client. This bounds the database read transaction to local snapshot creation instead of network transfer. The application owns the temporary file's cleanup.

Static imports and build tags

Import chosen drivers directly, or import _ "github.com/shibukawa/yuruidb/backends" with profiles:

Tags	Drivers
aws	s3, dynamodb
gcp	gcs, firestore, firebase
cloudflare	r2, d1, cfkv
individual driver names	the selected drivers

firebase also imports Firestore. Tags combine without duplicate registration. lambda alone selects no backend. go run -tags aws ./examples/drivers prints included drivers. Tags select imports, not runtime credentials or endpoints.

Extensions and testing

adapter.New wraps an atomic backend with record semantics; call Bind[T](config.Collection) before use. Optional ItemWriter, Scanner, Deleter, and CollectionTransport interfaces enable native tokens, value listing, physical deletion, and shared collection views. OpenRaw and Wrap(raw) retain the byte-oriented extension boundary.

Firebase hex-encodes physical IDs with a nonnumeric prefix to support names and byte ordering. SQL cursors/revisions cannot distinguish different injected databases unless a distinct Identity is supplied.

go test -race ./...
go vet ./...
(cd integration && go test -race ./...)
python3 scripts/check-builds.py

Both modules work without cgo. The integration module alone depends on the pure-Go modernc.org/sqlite and PostgreSQL lib/pq drivers. Tests exercise actual bbolt/SQLite and independent HTTP fixtures, including response loss, stale writers after recreation, multi-collection access, 2000-to-10 record deletion, cloud credential flows, compatibility probes, and 34 MiB streaming. Live PostgreSQL Blob tests run when YURUIDB_POSTGRES_DSN is set; see integration. Live cloud services, PostgreSQL record conformance, IAM, and production performance remain unverified. See the implementation guide and validation history.

Documentation

Overview

Package yuruidb provides JSON-typed, multi-collection record storage. Import a driver to register it, call Open, and Bind each data type.

Index

• Variables
• func DecodeConfig[T any](input any) (T, error)
• func Drivers() []string
• func NewID() (string, error)
• func NewRevision() (string, error)
• func OutcomeUnknown(cause error) error
• func Register(name string, factory Factory)
• func ValidateCollection(name string) error
• func ValidateID(id string) error
• type Capabilities
• type CollectionOpener
• type Config
• type Factory
• type ListOptions
• type MutationError
  • func (e *MutationError) Error() string
  • func (e *MutationError) MatchesRevision(value string) bool
  • func (e *MutationError) Unwrap() error
• type Page
• type RawPage
• type RawRecord
• type RawStore
  • func OpenRaw(ctx context.Context, name string, options any) (RawStore, error)
• type Record
  • func (r Record[T]) ID() string
  • func (r Record[T]) MarshalJSON() ([]byte, error)
  • func (r Record[T]) Revision() string
  • func (r *Record[T]) UnmarshalJSON(data []byte) error
• type Store
  • func Open(ctx context.Context, name string, options any) (*Store, error)
  • func Wrap(raw RawStore) *Store
  • func (s *Store) Backup(ctx context.Context, w io.Writer) error
  • func (s *Store) Bind[T any](collection string) error
  • func (s *Store) Capabilities() Capabilities
  • func (s *Store) Close() error
  • func (s *Store) Delete[T any](ctx context.Context, id, revision string) error
  • func (s *Store) Get[T any](ctx context.Context, id string) (Record[T], error)
  • func (s *Store) Insert[T any](ctx context.Context, id string, data T) (Record[T], error)
  • func (s *Store) List[T any](ctx context.Context, options ListOptions) (Page[T], error)
  • func (s *Store) PurgeTombstones[T any](ctx context.Context) (int, error)
  • func (s *Store) Remove[T any](ctx context.Context, id string) error
  • func (s *Store) Set[T any](ctx context.Context, id string, data T) (Record[T], error)
  • func (s *Store) Update[T any](ctx context.Context, id string, data T, revision string) (Record[T], error)

Variables

var (
	ErrNotFound      = errors.New("yuruidb: record not found")
	ErrAlreadyExists = errors.New("yuruidb: record already exists")
	ErrConflict      = errors.New("yuruidb: revision conflict")
	ErrUnsupported   = errors.New("yuruidb: unsupported capability")
	ErrInvalid       = errors.New("yuruidb: invalid argument")
	ErrClosed        = errors.New("yuruidb: store closed")
	// ErrOutcomeUnknown means a remote mutation may have committed. Do not retry
	// with a different ID or revision without reconciling the outcome.
	// Built-in drivers exclude ErrConflict/ErrAlreadyExists from unknown outcomes.
	ErrOutcomeUnknown = errors.New("yuruidb: mutation outcome unknown")
)

Functions

func DecodeConfig

func DecodeConfig[T any](input any) (T, error)

DecodeConfig helps statically imported third-party factories accept either their own configuration type or a driver-neutral Config/map. Unknown fields are rejected. It performs no I/O or environment lookup.

func Drivers

func Drivers() []string

func NewID

func NewID() (string, error)

NewID creates an RFC 9562 UUIDv7. IDs are monotonic within this process even if its wall clock moves backward. Independent processes are not coordinated.

func NewRevision

func NewRevision() (string, error)

func OutcomeUnknown

func OutcomeUnknown(cause error) error

OutcomeUnknown preserves diagnostic causes while excluding definitive conflict and duplicate classifications. A retry's failed condition cannot establish whether an earlier request committed. Other Is/As queries still reach cause.

func Register

func Register(name string, factory Factory)

Register is intended for driver init functions. Registration does no I/O.

func ValidateCollection

func ValidateCollection(name string) error

ValidateCollection accepts a portable, single-component collection name.

func ValidateID

func ValidateID(id string) error

ValidateID accepts a portable ASCII name of 1..256 bytes. Slash and URL metacharacters are excluded. IDs retain case and are ordered bytewise.

Types

type Capabilities

type Capabilities struct {
	ConditionalMutations bool
	Distributed          bool
	StrongReads          bool
}

Capabilities describes guarantees, independently of driver availability.

type CollectionOpener

type CollectionOpener interface {
	OpenCollection(context.Context, string) (RawStore, error)
}

CollectionOpener supplies borrowed collection views. Their lifetime is owned by the root RawStore; closing a view must not close the shared connection.

type Config

type Config map[string]any

Config is the driver-neutral configuration path for applications that import only backends and choose drivers through build tags. Keys are the fields of the selected driver's Config, case-insensitively. Typed constructors and typed driver Config values remain available for direct imports. Function and client values can be supplied without serializing credentials or callbacks.

type Factory

type Factory func(context.Context, any) (RawStore, error)

type ListOptions

type ListOptions struct {
	Limit  int
	Cursor string
}

type MutationError

type MutationError struct {
	Operation, Collection, ID, Nonce, ExpectedRevision, AttemptedRevision string
	Cause                                                                 error
}

MutationError preserves the identity of an attempted mutation for later reconciliation. Nonce is stored in the envelope; AttemptedRevision may omit a native token when the response was lost. Unwrap preserves diagnostic Is/As queries except definitive ErrConflict and ErrAlreadyExists classifications.

func (*MutationError) Error

func (e *MutationError) Error() string

func (*MutationError) MatchesRevision

func (e *MutationError) MatchesRevision(value string) bool

MatchesRevision reports whether a retrieved revision identifies this exact attempted write, ignoring a native token unavailable after response loss. False does not prove failure: the write may have been superseded or deleted.

func (*MutationError) Unwrap

func (e *MutationError) Unwrap() error

type Page

type Page[T any] struct {
	Records []Record[T]
	Cursor  string
}

type RawPage

type RawPage struct {
	Records []RawRecord
	Cursor  string
}

type RawRecord

type RawRecord struct {
	ID       string
	Value    []byte
	Revision string
}

RawRecord is the byte-oriented driver record. Value contains caller-owned bytes. ID is immutable; Revision is an opaque concurrency token.

type RawStore

type RawStore interface {
	Insert(context.Context, string, []byte) (RawRecord, error)
	Get(context.Context, string) (RawRecord, error)
	List(context.Context, ListOptions) (RawPage, error)
	Update(context.Context, string, []byte, string) (RawRecord, error)
	Delete(context.Context, string, string) error
	Capabilities() Capabilities
	Close() error
}

RawStore is the byte-oriented contract implemented by drivers. Values are opaque bytes at this layer; Store handles JSON encoding. Each store is bound to one collection. List traverses immutable IDs in ascending byte order, without page snapshots. Update and Delete require a previously retrieved revision.

func OpenRaw

func OpenRaw(ctx context.Context, name string, options any) (RawStore, error)

OpenRaw selects a statically imported driver. Options can be that driver's Config value or a driver-neutral yuruidb.Config map. No dynamic loading or implicit provider detection occurs.

type Record

type Record[T any] struct {
	Data T
	// contains filtered or unexported fields
}

Record holds JSON data and read-only storage metadata. Modify Data and pass it to Update with ID() and Revision(); the returned record has a new revision.

func (Record[T]) ID

func (r Record[T]) ID() string

ID returns the immutable record ID.

func (Record[T]) MarshalJSON

func (r Record[T]) MarshalJSON() ([]byte, error)

MarshalJSON includes metadata as id and revision alongside data.

func (Record[T]) Revision

func (r Record[T]) Revision() string

Revision returns the opaque concurrency token for this retrieved version. Updating storage returns a new Record without modifying the original value.

func (*Record[T]) UnmarshalJSON

func (r *Record[T]) UnmarshalJSON(data []byte) error

UnmarshalJSON replaces the complete record, restoring metadata from JSON. A decoding error leaves the existing record unchanged.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store owns a backend connection and binds distinct Go types to collections. Its generic methods encode JSON; Record[T] retains the requested data type. List uses ascending ID byte order and does not provide page snapshots.

func Open

func Open(ctx context.Context, name string, options any) (*Store, error)

Open opens one backend. Omit Collection in driver options and call Bind for each data type. Driver constructors expose the same connection-scoped API.

func Wrap

func Wrap(raw RawStore) *Store

Wrap gives ownership of a raw backend to a JSON store. Bind each type before use. A third-party raw backend without OpenCollection can bind one collection.

func (*Store) Backup

func (s *Store) Backup(ctx context.Context, w io.Writer) error

Backup writes a consistent snapshot of the entire backend, including all collections. Currently supported by bbolt. The caller owns and closes w.

func (*Store) Bind

func (s *Store) Bind[T any](collection string) error

Bind registers a stable collection name for a data type. Defined types such as type Done Job and type Waiting Job are distinct; aliases are not. Pointer and value forms resolve to the same collection. Registration does no I/O.

func (*Store) Capabilities

func (s *Store) Capabilities() Capabilities

func (*Store) Close

func (s *Store) Close() error

func (*Store) Delete

func (s *Store) Delete[T any](ctx context.Context, id, revision string) error

func (*Store) Get

func (s *Store) Get[T any](ctx context.Context, id string) (Record[T], error)

func (*Store) Insert

func (s *Store) Insert[T any](ctx context.Context, id string, data T) (Record[T], error)

Insert creates an absent ID. It never silently replaces an existing record.

func (*Store) List

func (s *Store) List[T any](ctx context.Context, options ListOptions) (Page[T], error)

func (*Store) PurgeTombstones

func (s *Store) PurgeTombstones[T any](ctx context.Context) (int, error)

PurgeTombstones removes retained tombstones using conditional deletion.

func (*Store) Remove

func (s *Store) Remove[T any](ctx context.Context, id string) error

Remove unconditionally deletes an ID on drivers with explicit support for this operation, currently Cloudflare KV. Other drivers return ErrUnsupported.

func (*Store) Set

func (s *Store) Set[T any](ctx context.Context, id string, data T) (Record[T], error)

Set unconditionally creates or replaces data on drivers that explicitly support this operation, currently Cloudflare KV. Revision cannot guard it. Other drivers return ErrUnsupported; this is never a fallback for Update.

func (*Store) Update

func (s *Store) Update[T any](ctx context.Context, id string, data T, revision string) (Record[T], error)

Update replaces Data while preserving ID() and its position in List.