iodb

package module
v0.4.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 31, 2026 License: MIT Imports: 10 Imported by: 0

README

iodb   GoDoc Coverage Go Report Card MIT licensed

banner

A lightweight, file-system-backed key/value store for Go.

iodb maps:

  • Buckets to directories
  • Files to values

It is useful when you want simple persistent storage without running a separate database process.

Under the hood, iodb uses streambuf for file read/write behavior.

  • Independent readers with their own cursors
  • Tail-style streaming reads (StreamingRead)
  • Safe concurrent append and read coordination

Examples

New
func ExampleNew() {
	var err error
	if exampleDB, err = New("path/to/dir"); err != nil {
		log.Fatal(err)
	}
}
Bucket.GetBucket
func ExampleBucket_GetBucket() {
	var (
		b  *Bucket
		ok bool
	)

	if b, ok = exampleBucket.GetBucket("my_bucket"); !ok {
		log.Fatalf("my_bucket not found")
	}

	fmt.Println("Bucket", b)
}
Bucket.CreateBucket
func ExampleBucket_CreateBucket() {
	var (
		b   *Bucket
		err error
	)

	if b, err = exampleBucket.CreateBucket("my_bucket"); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Bucket", b)
}
Bucket.GetOrCreateBucket
func ExampleBucket_GetOrCreateBucket() {
	var (
		b   *Bucket
		err error
	)

	if b, err = exampleBucket.GetOrCreateBucket("my_bucket"); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Bucket", b)
}
Bucket.Get
func ExampleBucket_Get() {
	var (
		f  *File
		ok bool
	)

	if f, ok = exampleBucket.Get("my_file"); !ok {
		log.Fatalf("my_file not found")
	}

	fmt.Println("File", f)
}
Bucket.Create
func ExampleBucket_Create() {
	var (
		f   *File
		err error
	)

	if f, err = exampleBucket.Create("my_file"); err != nil {
		log.Fatal(err)
	}

	fmt.Println("File", f)
}
Bucket.GetOrCreate
func ExampleBucket_GetOrCreate() {
	var (
		f   *File
		err error
	)

	if f, err = exampleBucket.GetOrCreate("my_file"); err != nil {
		log.Fatal(err)
	}

	fmt.Println("File", f)
}
Bucket.Delete
func ExampleBucket_Delete() {
	var err error
	if err = exampleBucket.Delete("my_file"); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Deleted my_file")
}
Bucket.Cursor
func ExampleBucket_Cursor() {
	var err error
	if err = exampleBucket.Cursor(func(c *Cursor) error {
		var (
			f  *File
			ok bool
		)

		if f, ok = c.First(); !ok {
			return errors.New("empty bucket")
		}

		fmt.Println("First file", f)
		return nil
	}); err != nil {
		log.Fatal(err)
	}
}
Bucket.ForEach
func ExampleBucket_ForEach() {
	var err error
	if err = exampleBucket.ForEach(func(f *File) error {
		fmt.Println("File", f)
		return nil
	}); err != nil {
		log.Fatal(err)
	}
}
File.Read
func ExampleFile_Read() {
	var err error
	if err = exampleFile.Read(func(r io.Reader) (err error) {
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Read success")
}
File.StreamingRead
func ExampleFile_StreamingRead() {
	var err error
	if err = exampleFile.StreamingRead(context.Background(), func(r io.Reader) (err error) {
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	fmt.Println("StreamingRead success")
}
File.Update
func ExampleFile_Update() {
	var err error
	if err = exampleFile.Update(func(w io.Writer) (err error) {
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Update success")
}
File.Append
func ExampleFile_Append() {
	var err error
	if err = exampleFile.Append(func(w io.Writer) (err error) {
		return nil
	}); err != nil {
		log.Fatal(err)
	}

	fmt.Println("Append success")
}

Data Model

  • DB is the root bucket.
  • Bucket represents a directory.
  • File represents one persisted value inside a bucket.

API Overview

Database
  • New(path string) (*DB, error)
    • Creates the root directory if missing.
    • Loads existing files and child buckets from disk.
Bucket
  • GetBucket(key string) (*Bucket, bool)
  • CreateBucket(key string) (*Bucket, error)
  • GetOrCreateBucket(key string) (*Bucket, error)
  • Get(key string) (*File, bool)
  • Create(key string) (*File, error)
  • GetOrCreate(key string) (*File, error)
  • Delete(key string) error
  • Cursor(func(*Cursor) error) error
  • ForEach(func(*File) error) error
File
  • Read(func(io.Reader) error) error
    • Non-streaming read of the current buffer state.
  • StreamingRead(ctx context.Context, func(io.Reader) error) error
    • Tail-style streaming read.
  • Update(func(io.Writer) error) error
    • Atomic replace using temp file + rename + directory sync.
  • Append(func(io.Writer) error) error
    • Appends to the current stream buffer.

Key Rules

Keys are validated for file and bucket creation:

  • Must not be empty
  • Must match ^[A-Za-z0-9](?:[A-Za-z0-9._-]{0,254})$
  • Path separators are not allowed
  • Maximum length is 255 characters

Exported validation errors:

  • ErrEmptyKey
  • ErrInvalidKeyFormat

Temporary-file naming:

  • The .tmp_ prefix is reserved for internal iodb temp files.
  • During bucket load, files prefixed with .tmp_ are treated as stale temp artifacts and removed.

Concurrency and Update Semantics

  • Bucket lookups/creates are guarded by sync.RWMutex.
  • File updates are synchronized with a transaction mutex.
  • Update rotates the backing stream buffer after replacing the on-disk file.
  • Readers started before rotation may continue on the previous buffer.
  • Appends racing with Update can return streambuf.ErrIsClosed.
  • Read is snapshot-style and reaches EOF at current end.
  • StreamingRead is tail-style and waits for new bytes until closed/canceled.
  • Bucket.Cursor and Bucket.ForEach execute callbacks while holding the bucket read lock; callbacks must not call mutating methods on that same bucket (CreateBucket, GetOrCreateBucket, Create, GetOrCreate, or Delete) to avoid deadlocks.

Testing

go test --race

AI Usage and Authorship

This project is intentionally human-authored for all logic.

To be explicit:

  • AI does not write or modify non-test code in this repository.
  • AI does not make architectural or behavioral decisions.
  • AI may assist with documentation, comments, and test scaffolding only.
  • All implementation logic is written and reviewed by human maintainers.

These boundaries are enforced in AGENTS.md and are part of this repository's contribution discipline.

Contributors

  • Human maintainers: library design, implementation, and behavior decisions.
  • ChatGPT Codex: documentation, test coverage support, and comments.
  • Google Gemini: README artwork generation.

License

MIT

footer

Documentation

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrEmptyKey is returned when a key argument is an empty string.
	ErrEmptyKey = errors.New("invalid key, cannot be empty")
	// ErrInvalidKeyFormat is returned when a key contains disallowed characters or separators.
	ErrInvalidKeyFormat = errors.New("invalid key format: only letters, digits, '.', '_', '-' are allowed; key must not contain path separators")
)

Functions

This section is empty.

Types

type Bucket 

type Bucket struct {
	// contains filtered or unexported fields
}

Bucket groups child buckets and files under a shared directory path.

func (*Bucket) Create 

func (b *Bucket) Create(key string) (out *File, err error)

Create validates key and creates a file when it is not already present.

Example 
var (
	f   *File
	err error
)

if f, err = exampleBucket.Create("my_file"); err != nil {
	log.Fatal(err)
}

fmt.Println("File", f)

func (*Bucket) CreateBucket 

func (b *Bucket) CreateBucket(key string) (out *Bucket, err error)

CreateBucket validates key and creates a child bucket when missing.

Example 
var (
	b   *Bucket
	err error
)

if b, err = exampleBucket.CreateBucket("my_bucket"); err != nil {
	log.Fatal(err)
}

fmt.Println("Bucket", b)

func (*Bucket) Cursor added in v0.2.0 

func (b *Bucket) Cursor(fn func(*Cursor) error) (err error)

Cursor executes fn with a read-only cursor over files in key order.

fn executes while the bucket read lock is held.

Callbacks must not call methods that may require a write lock on this same bucket (for example: CreateBucket, GetOrCreateBucket, Create, GetOrCreate, or Delete), because doing so can deadlock.

The cursor is only valid for the duration of fn.

Example 
var err error
if err = exampleBucket.Cursor(func(c *Cursor) error {
	var (
		f  *File
		ok bool
	)

	if f, ok = c.First(); !ok {
		return errors.New("empty bucket")
	}

	fmt.Println("First file", f)
	return nil
}); err != nil {
	log.Fatal(err)
}

func (*Bucket) Delete added in v0.3.0 

func (b *Bucket) Delete(key string) (err error)

Delete removes key from the bucket if present.

Delete validates key, closes the file buffer when initialized, removes the file from disk, and then removes the key from the in-memory index.

If key is not present, Delete returns nil.

Example 
var err error
if err = exampleBucket.Delete("my_file"); err != nil {
	log.Fatal(err)
}

fmt.Println("Deleted my_file")

func (*Bucket) ForEach added in v0.2.0 

func (b *Bucket) ForEach(fn func(*File) error) (err error)

ForEach calls fn for each file in key order until fn returns an error.

fn executes while the bucket read lock is held.

Callbacks must not call methods that may require a write lock on this same bucket (for example: CreateBucket, GetOrCreateBucket, Create, GetOrCreate, or Delete), because doing so can deadlock.

Example 
var err error
if err = exampleBucket.ForEach(func(f *File) error {
	fmt.Println("File", f)
	return nil
}); err != nil {
	log.Fatal(err)
}

func (*Bucket) Get 

func (b *Bucket) Get(key string) (out *File, ok bool)

Get returns the file for key when it exists.

Example 
var (
	f  *File
	ok bool
)

if f, ok = exampleBucket.Get("my_file"); !ok {
	log.Fatalf("my_file not found")
}

fmt.Println("File", f)

func (*Bucket) GetBucket 

func (b *Bucket) GetBucket(key string) (out *Bucket, ok bool)

GetBucket returns the child bucket for key when it exists.

Example 
var (
	b  *Bucket
	ok bool
)

if b, ok = exampleBucket.GetBucket("my_bucket"); !ok {
	log.Fatalf("my_bucket not found")
}

fmt.Println("Bucket", b)

func (*Bucket) GetOrCreate 

func (b *Bucket) GetOrCreate(key string) (out *File, err error)

GetOrCreate returns an existing file for key or creates it.

Example 
var (
	f   *File
	err error
)

if f, err = exampleBucket.GetOrCreate("my_file"); err != nil {
	log.Fatal(err)
}

fmt.Println("File", f)

func (*Bucket) GetOrCreateBucket 

func (b *Bucket) GetOrCreateBucket(key string) (out *Bucket, err error)

GetOrCreateBucket returns an existing child bucket or creates it.

Example 
var (
	b   *Bucket
	err error
)

if b, err = exampleBucket.GetOrCreateBucket("my_bucket"); err != nil {
	log.Fatal(err)
}

fmt.Println("Bucket", b)

func (Bucket) Key 

func (e Bucket) Key() (out string)

Key returns the entry key used as the on-disk file or directory name.

type Cursor added in v0.2.0 

type Cursor struct {
	*bst.Cursor[*File]
}

Cursor iterates files in key order for a bucket snapshot.

type DB 

type DB struct {
	*Bucket
}

DB is the root container for buckets and files under a database path.

func New 

func New(dbPath string) (out *DB, err error)

New creates or opens a database rooted at dbPath.

Example 
var err error
if exampleDB, err = New("path/to/dir"); err != nil {
	log.Fatal(err)
}

func (DB) Key 

func (e DB) Key() (out string)

Key returns the entry key used as the on-disk file or directory name.

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents a single persisted value in the database.

func (*File) Append 

func (f *File) Append(fn func(io.Writer) error) (err error)

Append writes to the current stream buffer via fn.

During a concurrent Update, append operations may fail with streambuf.ErrIsClosed if they target a buffer being rotated out.

Example 
var err error
if err = exampleFile.Append(func(w io.Writer) (err error) {
	return nil
}); err != nil {
	log.Fatal(err)
}

fmt.Println("Append success")

func (File) Key 

func (e File) Key() (out string)

Key returns the entry key used as the on-disk file or directory name.

func (*File) Read 

func (f *File) Read(fn func(io.Reader) error) (err error)

Read opens a non-streaming reader for the current stream buffer and passes it to fn. The reader is closed after fn returns.

If Read acquires a reader before or during Update, it may continue reading from the previous buffer.

Reads that start after Update returns observe the updated file contents.

Example 
var err error
if err = exampleFile.Read(func(r io.Reader) (err error) {
	return nil
}); err != nil {
	log.Fatal(err)
}

fmt.Println("Read success")

func (*File) StreamingRead 

func (f *File) StreamingRead(ctx context.Context, fn func(io.Reader) error) (err error)

StreamingRead opens a tail-style streaming reader for the current stream buffer and passes it to fn.

The reader is closed when fn returns or when ctx is canceled, whichever happens first.

While active, the reader can observe new data appended through Append against the same active buffer.

Like Read, if StreamingRead acquires a reader before or during Update, it may continue reading from the previous buffer according to streambuf close semantics. It does not automatically move to the new buffer created by Update.

When no bytes are available and the streaming reader or backing buffer is closed, streambuf returns streambuf.ErrIsClosed.

Example 
var err error
if err = exampleFile.StreamingRead(context.Background(), func(r io.Reader) (err error) {
	return nil
}); err != nil {
	log.Fatal(err)
}

fmt.Println("StreamingRead success")

func (*File) Update 

func (f *File) Update(fn func(io.Writer) error) (err error)

Update writes file contents through fn and atomically replaces the file.

Update replaces the on-disk file via rename, syncs the parent directory for durability, then rotates the in-memory stream buffer.

While rotation is in progress, in-flight readers on the previous buffer continue using that buffer. Non-streaming readers drain available bytes and then reach EOF-equivalent behavior, while streaming readers unblock and end according to streambuf close semantics. Appends using a buffer being closed may fail with streambuf.ErrIsClosed.

On successful return from Update, newly started Read and StreamingRead calls observe updated contents. In-flight reads and appends may still complete against the previous buffer according to streambuf close semantics.

Example 
var err error
if err = exampleFile.Update(func(w io.Writer) (err error) {
	return nil
}); err != nil {
	log.Fatal(err)
}

fmt.Println("Update success")

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.