storm

package module
v2.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: MIT Imports: 31 Imported by: 0

README

Storm-rev

Go Reference

This project is a fork of storm, which is a simple and powerful toolkit for BoltDB.

Chinese documentation: README.zh-CN.md.

Storm-rev is a simple and powerful toolkit for BoltDB. It provides typed CRUD helpers, external Bleve indexes, composite indexes, full-text search, an advanced query system, and much more.

In addition to the examples below, see also the examples in the GoDoc.

For extended queries and support for Badger, see also Genji

Table of Contents

Getting Started

GO111MODULE=on go get -u github.com/clakeboy/storm-rev

Import Storm

import "github.com/clakeboy/storm-rev"

Open a database

Quick way of opening a database

db, err := storm.Open("my.db")

defer db.Close()

Open can receive multiple options to customize the way it behaves. See Options below

Simple CRUD system

Declare your structures
type User struct {
  ID int // primary key
  Group string `storm:"index,composite=group_age:1"` // indexed and used as the first field of a composite index
  Email string `storm:"unique"` // indexed with a unique constraint
  Name string // this field will not be indexed
  Age int `storm:"index,composite=group_age:2"` // indexed and used as the second field of a composite index
  Bio string `storm:"fulltext"` // indexed for full-text search
}

The primary key can be of any type as long as it is not a zero value. Storm will search for the tag id, if not present Storm will search for a field named ID.

type User struct {
  ThePrimaryKey string `storm:"id"`// primary key
  Group string `storm:"index"` // this field will be indexed
  Email string `storm:"unique"` // this field will be indexed with a unique constraint
  Name string // this field will not be indexed
}

Storm handles tags in nested structures with the inline tag

type Base struct {
  Ident bson.ObjectId `storm:"id"`
}

type User struct {
  Base      `storm:"inline"`
  Group     string `storm:"index"`
  Email     string `storm:"unique"`
  Name      string
  CreatedAt time.Time `storm:"index"`
}

Supported index tags:

  • storm:"index": single-field exact/range/prefix index.
  • storm:"unique": single-field index with a unique constraint.
  • storm:"fulltext": full-text index for Search.
  • storm:"composite=name:order": adds a field to a named composite index. Orders must start at 1 and be continuous.
  • storm:"id": primary key. If omitted, a field named ID is used.
  • storm:"increment": auto-increment integer field.
  • storm:"inline": reads tags from an embedded struct.
Save your object
user := User{
  ID: 10,
  Group: "staff",
  Email: "john@provider.com",
  Name: "John",
  Age: 21,
  CreatedAt: time.Now(),
}

err := db.Save(&user)
// err == nil

user.ID++
err = db.Save(&user)
// err == storm.ErrAlreadyExists

That's it.

Save creates or updates all the required indexes and buckets, checks the unique constraints and saves the object to the store.

Use SaveAll when importing many objects. It validates and writes the whole slice in one BoltDB write transaction, then updates the external indexes in batches.

users := []User{
  {ID: 1, Name: "John"},
  {ID: 2, Name: "Jane"},
}

err := db.SaveAll(users)
Auto Increment

Storm can auto increment integer values so you don't have to worry about that when saving your objects. Also, the new value is automatically inserted in your field.


type Product struct {
  Pk                  int `storm:"id,increment"` // primary key with auto increment
  Name                string
  IntegerField        uint64 `storm:"increment"`
  IndexedIntegerField uint32 `storm:"index,increment"`
  UniqueIntegerField  int16  `storm:"unique,increment=100"` // the starting value can be set
}

p := Product{Name: "Vaccum Cleaner"}

fmt.Println(p.Pk)
fmt.Println(p.IntegerField)
fmt.Println(p.IndexedIntegerField)
fmt.Println(p.UniqueIntegerField)
// 0
// 0
// 0
// 0

_ = db.Save(&p)

fmt.Println(p.Pk)
fmt.Println(p.IntegerField)
fmt.Println(p.IndexedIntegerField)
fmt.Println(p.UniqueIntegerField)
// 1
// 1
// 1
// 100
Simple queries

Any object can be fetched, indexed or not. Storm uses indexes when available, otherwise it uses the query system.

Fetch one object
var user User
err := db.One("Email", "john@provider.com", &user)
// err == nil

err = db.One("Name", "John", &user)
// err == nil

err = db.One("Name", "Jack", &user)
// err == storm.ErrNotFound
Fetch multiple objects
var users []User
err := db.Find("Group", "staff", &users)
Fetch all objects
var users []User
err := db.All(&users)
Fetch all objects sorted by index
var users []User
err := db.AllByIndex("CreatedAt", &users)
Fetch a range of objects
var users []User
err := db.Range("Age", 10, 21, &users)
Fetch objects by prefix
var users []User
err := db.Prefix("Name", "Jo", &users)
Fetch objects by composite index

Composite indexes are declared with composite=<index name>:<field order> and queried with FindByIndex. The current composite index API supports full equality matches.

type User struct {
  ID    int
  Group string `storm:"index,composite=group_age:1"`
  Age   int    `storm:"index,composite=group_age:2"`
}

var users []User
err := db.FindByIndex("group_age", []any{"staff", 21}, &users)

Select can also use a composite index when an And matcher contains equality conditions for every field in the composite index:

err = db.Select(q.And(
  q.Eq("Group", "staff"),
  q.Eq("Age", 21),
)).Find(&users)

In does not currently expand into composite-index lookups. A query such as q.And(q.In("Group", []string{"staff", "admin"}), q.Eq("Age", 21)) can still use a single-field index when one exists, then the full matcher filters the candidate records. To force composite-index usage, expand the condition into Or branches where each branch has full equality values:

err = db.Select(q.Or(
  q.And(q.Eq("Group", "staff"), q.Eq("Age", 21)),
  q.And(q.Eq("Group", "admin"), q.Eq("Age", 21)),
)).Find(&users)
Full-text search

Use storm:"fulltext" for fields that should be analyzed by Bleve and queried with Search. This is separate from Find, which keeps exact-match semantics.

type Article struct {
  ID    int
  Title string `storm:"fulltext"`
}

var articles []Article
err := db.Search("Title", "bleve search", &articles)
Skip, Limit and Reverse
var users []User
err := db.Find("Group", "staff", &users, storm.Skip(10))
err = db.Find("Group", "staff", &users, storm.Limit(10))
err = db.Find("Group", "staff", &users, storm.Reverse())
err = db.Find("Group", "staff", &users, storm.Limit(10), storm.Skip(10), storm.Reverse())

err = db.All(&users, storm.Limit(10), storm.Skip(10), storm.Reverse())
err = db.AllByIndex("CreatedAt", &users, storm.Limit(10), storm.Skip(10), storm.Reverse())
err = db.Range("Age", 10, 21, &users, storm.Limit(10), storm.Skip(10), storm.Reverse())
err = db.Search("Title", "bleve", &articles, storm.Limit(10), storm.Skip(10), storm.Reverse())
Delete an object
err := db.DeleteStruct(&user)
Update an object
// Update multiple fields
// Only works for non zero-value fields (e.g. Name can not be "", Age can not be 0)
err := db.Update(&User{ID: 10, Name: "Jack", Age: 45})

// Update a single field
// Also works for zero-value fields (0, false, "", ...)
err := db.UpdateField(&User{ID: 10}, "Age", 0)
Initialize buckets and indexes before saving an object
err := db.Init(&User{})

Useful when starting your application

Drop a bucket

Using the struct

err := db.Drop(&User)

Using the bucket name

err := db.Drop("User")
Re-index a bucket
err := db.ReIndex(&User{})

Useful when the structure has changed, when external index files need to be rebuilt, or after an index was marked dirty because an index update or consistency check failed. ReIndex is the only public index rebuild method.

Index files

Storm-rev stores records in BoltDB and stores indexes as external Bleve indexes. If the Bolt database is /path/app.db, index files are stored under /path/app_db_index/. Each table gets its own Bleve index directory, for example /path/app_db_index/User.bleve.

BoltDB remains the source of truth. Bleve indexes are derived data and can be recreated from BoltDB at any time. The index documents store exact-match tokens, typed values for range/prefix scans, field-existence markers, full-text fields, and composite-index markers for the indexed struct fields.

Indexed writes append a durable, sequence-ordered outbox entry in the same BoltDB transaction as the record change. After that transaction commits, one bounded coordinator combines concurrent work into Bleve batches. This keeps Bleve work outside Bolt's writer lock, prevents concurrent Update calls from creating an unbounded segment storm, and lets startup replay any committed index work that was not applied before a crash. Updates that do not change an indexed value skip Bleve entirely.

Index writes are asynchronous by default: mutating APIs return after the Bolt transaction and durable outbox commit. Use FlushBleve(ctx) when an explicit index-visibility barrier is needed, or open the database with BleveSyncWrites() when every mutation must wait for Bleve and report Bleve errors directly. The coordinator queue is bounded, so asynchronous writers can still receive backpressure when Bleve cannot keep up. BleveAsyncWrites() remains available as an explicit declaration of the default behavior.

If an index hit points to a missing Bolt record, or an index update fails, the table index is marked dirty and indexed queries safely fall back to Bolt scans. Failed outbox entries are retried in sequence; repeated failures trigger an automatic rebuild. Pending entries and dirty state survive restart. Callers can also rebuild explicitly with:

err := db.ReIndex(&User{})

ReIndex reads a Bolt snapshot and builds a temporary Bleve index in bounded batches, then atomically switches the index directory. It does not hold Bolt's writer lock during the rebuild. Indexed writes committed while the snapshot is being rebuilt remain in the outbox and are replayed after the switch.

Advanced queries

For more complex queries, you can use the Select method. Select takes any number of Matcher from the q package.

Select uses external indexes when it can build a safe candidate plan, then still runs the original matcher as the final filter. It can use Eq/StrictEq on indexed fields and IDs, In on indexed fields, closed ranges written as And(Gte(...), Lte(...)), complete composite equality matches, and Or only when every branch can use an index. Unsupported matchers, unindexed conditions, zero/nil values that are not present in the index, open transactions, and dirty indexes fall back to scanning the Bolt bucket. OrderBy, Skip, Limit, Reverse, Find, First, Count, Each, Raw, and Delete continue to use the same query pipeline.

Here are some common Matchers:

// Equality
q.Eq("Name", John)

// Strictly greater than
q.Gt("Age", 7)

// Lesser than or equal to
q.Lte("Age", 77)

// Regex with name that starts with the letter D
q.Re("Name", "^D")

// In the given slice of values
q.In("Group", []string{"Staff", "Admin"})

// Comparing fields
q.EqF("FieldName", "SecondFieldName")
q.LtF("FieldName", "SecondFieldName")
q.GtF("FieldName", "SecondFieldName")
q.LteF("FieldName", "SecondFieldName")
q.GteF("FieldName", "SecondFieldName")

Matchers can also be combined with And, Or and Not:


// Match if all match
q.And(
  q.Gt("Age", 7),
  q.Re("Name", "^D")
)

// Match if one matches
q.Or(
  q.Re("Name", "^A"),
  q.Not(
    q.Re("Name", "^B")
  ),
  q.Re("Name", "^C"),
  q.In("Group", []string{"Staff", "Admin"}),
  q.And(
    q.StrictEq("Password", []byte(password)),
    q.Eq("Registered", true)
  )
)

You can find the complete list in the documentation.

Select takes any number of matchers and wraps them into a q.And() so it's not necessary to specify it. It returns a Query type.

query := db.Select(q.Gte("Age", 7), q.Lte("Age", 77))

The Query type contains methods to filter and order the records.

// Limit
query = query.Limit(10)

// Skip
query = query.Skip(20)

// Calls can also be chained
query = query.Limit(10).Skip(20).OrderBy("Age").Reverse()

But also to specify how to fetch them.

var users []User
err = query.Find(&users)

var user User
err = query.First(&user)

Examples with Select:

// Find all users with an ID between 10 and 100
err = db.Select(q.Gte("ID", 10), q.Lte("ID", 100)).Find(&users)

// Nested matchers
err = db.Select(q.Or(
  q.Gt("ID", 50),
  q.Lt("Age", 21),
  q.And(
    q.Eq("Group", "admin"),
    q.Gte("Age", 21),
  ),
)).Find(&users)

query := db.Select(q.Gte("ID", 10), q.Lte("ID", 100)).Limit(10).Skip(5).Reverse().OrderBy("Age", "Name")

// Find multiple records
err = query.Find(&users)
// or
err = db.Select(q.Gte("ID", 10), q.Lte("ID", 100)).Limit(10).Skip(5).Reverse().OrderBy("Age", "Name").Find(&users)

// Find first record
err = query.First(&user)
// or
err = db.Select(q.Gte("ID", 10), q.Lte("ID", 100)).Limit(10).Skip(5).Reverse().OrderBy("Age", "Name").First(&user)

// Delete all matching records
err = query.Delete(new(User))

// Fetching records one by one (useful when the bucket contains a lot of records)
query = db.Select(q.Gte("ID", 10),q.Lte("ID", 100)).OrderBy("Age", "Name")

err = query.Each(new(User), func(record interface{}) error) {
  u := record.(*User)
  ...
  return nil
}

See the documentation for a complete list of methods.

SQL usage

Storm-rev also exposes a small SQL layer for single-table reads and writes. It is useful when callers already build SQL strings, while still keeping BoltDB as the source of truth and keeping Storm-rev indexes in sync.

Register model metadata explicitly when you create the SQL helper:

type User struct {
  ID     int    `storm:"id,increment" json:"id"`
  Name   string `storm:"index" json:"name"`
  Age    int    `storm:"index" json:"age"`
  Team   string `storm:"index" json:"team"`
  Active bool   `json:"active"`
}

// Register the User table and its columns.
sqlDB, err := db.SQL(&User{})
if err != nil {
  return err
}

The table name is the Storm bucket/model name. Column names can use either Go field names or JSON tag names, case-insensitively. If schema metadata already exists, for example after Init, Save, or SaveAll, db.SQL() can also load table definitions from metadata.

Use Find or First for SELECT * queries:

var users []User

// Placeholders are bound from the variadic args.
err := sqlDB.Find(
  "SELECT * FROM User WHERE age >= ? AND team IN (?, ?) ORDER BY age DESC LIMIT 10",
  &users,
  18,
  "staff",
  "admin",
)

Use Project for selected columns. The destination can be []map[string]any or a DTO slice:

type UserDTO struct {
  Username string `json:"username"`
  UserAge  int    `json:"user_age"`
}

var rows []UserDTO

// Column aliases are matched against map keys or DTO json tags.
err := sqlDB.Project(
  "SELECT name AS username, age AS user_age FROM User WHERE team = ? ORDER BY age ASC",
  &rows,
  "staff",
)

Use Count for SELECT COUNT(*):

count, err := sqlDB.Count(
  "SELECT COUNT(*) FROM User WHERE active = ?",
  true,
)

Use Exec for INSERT, UPDATE, and DELETE. Writes go through Storm-rev's save/delete path, so normal indexes and unique constraints are maintained.

result, err := sqlDB.Exec(
  "INSERT INTO User (name, age, team, active) VALUES (?, ?, ?, ?)",
  "Alice",
  31,
  "staff",
  true,
)
if err != nil {
  return err
}
fmt.Println(result.RowsAffected, result.LastInsertID)

result, err = sqlDB.Exec(
  "UPDATE User SET team = ?, age = ? WHERE name = ?",
  "ops",
  32,
  "Alice",
)

result, err = sqlDB.Exec(
  "DELETE FROM User WHERE active = ?",
  false,
)

For safety, UPDATE and DELETE without WHERE return ErrSQLUnsafeWrite by default. Use WithAllowFullTableWrite(true) only when a whole-table write is intentional:

result, err = sqlDB.WithAllowFullTableWrite(true).Exec("DELETE FROM User")

Supported SQL is intentionally limited:

  • One table per statement; joins, subqueries, GROUP BY, HAVING, and DISTINCT are not supported.
  • WHERE supports =, !=, >, >=, <, <=, IN, NOT IN, AND, OR, NOT, parentheses, literals, booleans, NULL, and ? placeholders.
  • ORDER BY supports one or more columns, but all columns must use the same direction.
  • LIMIT/offset are supported for SELECT.
  • Find and First require SELECT *; use Project for column lists and Count for COUNT(*).
  • INSERT requires an explicit column list and VALUES.
  • UPDATE does not support updating the ID field, ORDER BY, or LIMIT.
  • DELETE does not support ORDER BY or LIMIT.
Transactions
tx, err := db.Begin(true)
if err != nil {
  return err
}
defer tx.Rollback()

accountA.Amount -= 100
accountB.Amount += 100

err = tx.Save(accountA)
if err != nil {
  return err
}

err = tx.Save(accountB)
if err != nil {
  return err
}

return tx.Commit()

Queries inside an open transaction read from BoltDB and fall back to scanning instead of trusting external indexes, so they can see uncommitted changes. On Commit, pending index mutations are written to the durable outbox in the same Bolt transaction and applied after commit; tables explicitly marked for reindexing are then rebuilt from a read snapshot. Rollback discards both the BoltDB changes and the corresponding outbox work.

Writable transactions acquire the index commit-order gate before opening the Bolt write transaction. Commit, Rollback, and a failed Begin(true) release that gate, preventing an ordinary concurrent Save from holding the index gate while waiting for the explicit transaction's Bolt writer.

Options

Storm options are functions that can be passed when constructing you Storm instance. You can pass it any number of options.

Bleve write mode

External-index writes are asynchronous by default:

db, err := storm.Open("my.db")
if err != nil {
  return err
}

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = db.FlushBleve(ctx)

FlushBleve waits for both the durable outbox and the in-memory coordinator queue to drain. Close also drains healthy queued work; work that still fails remains in the Bolt outbox and is recovered on the next Open.

Use synchronous mode only where the write call itself must wait for Bleve visibility or return a Bleve failure:

db, err := storm.Open("my.db", storm.BleveSyncWrites())
BoltOptions

By default, Storm opens a database with the mode 0600 and a timeout of one second. You can change this behavior by using BoltOptions

db, err := storm.Open("my.db", storm.BoltOptions(0600, &bolt.Options{Timeout: 1 * time.Second}))
MarshalUnmarshaler

To store the data in BoltDB, Storm marshals it with Sonic-backed JSON by default. If you wish to change this behavior you can pass a codec that implements codec.MarshalUnmarshaler via the storm.Codec option:

db := storm.Open("my.db", storm.Codec(myCodec))
Provided Codecs

You can easily implement your own MarshalUnmarshaler, but Storm comes with built-in support for JSON (Sonic-backed by default, with the standard-library codec also available), GOB, Sereal, Protocol Buffers and MessagePack.

These can be used by importing the relevant package and use that codec to configure Storm. The example below shows all variants (without proper error handling):

import (
  "github.com/clakeboy/storm-rev"
  "github.com/clakeboy/storm-rev/codec/gob"
  "github.com/clakeboy/storm-rev/codec/json"
  "github.com/clakeboy/storm-rev/codec/sereal"
  "github.com/clakeboy/storm-rev/codec/protobuf"
  "github.com/clakeboy/storm-rev/codec/msgpack"
)

var gobDb, _ = storm.Open("gob.db", storm.Codec(gob.Codec))
var jsonDb, _ = storm.Open("json.db", storm.Codec(json.Codec))
var serealDb, _ = storm.Open("sereal.db", storm.Codec(sereal.Codec))
var protobufDb, _ = storm.Open("protobuf.db", storm.Codec(protobuf.Codec))
var msgpackDb, _ = storm.Open("msgpack.db", storm.Codec(msgpack.Codec))

Tip: Adding Storm tags to generated Protobuf files can be tricky. A good solution is to use this tool to inject the tags during the compilation.

Use existing Bolt connection

You can use an existing connection and pass it to Storm

bDB, _ := bolt.Open(filepath.Join(dir, "bolt.db"), 0600, &bolt.Options{Timeout: 10 * time.Second})
db := storm.Open("my.db", storm.UseDB(bDB))
Batch mode

Batch mode can be enabled to speed up concurrent writes (see Batch read-write transactions)

db := storm.Open("my.db", storm.Batch())

Nodes and nested buckets

Storm takes advantage of BoltDB nested buckets feature by using storm.Node. A storm.Node is the underlying object used by storm.DB to manipulate a bucket. To create a nested bucket and use the same API as storm.DB, you can use the DB.From method.

repo := db.From("repo")

err := repo.Save(&Issue{
  Title: "I want more features",
  Author: user.ID,
})

err = repo.Save(newRelease("0.10"))

var issues []Issue
err = repo.Find("Author", user.ID, &issues)

var release Release
err = repo.One("Tag", "0.10", &release)

You can also chain the nodes to create a hierarchy

chars := db.From("characters")
heroes := chars.From("heroes")
enemies := chars.From("enemies")

items := db.From("items")
potions := items.From("consumables").From("medicine").From("potions")

You can even pass the entire hierarchy as arguments to From:

privateNotes := db.From("notes", "private")
workNotes :=  db.From("notes", "work")
Node options

A Node can also be configured. Activating an option on a Node creates a copy, so a Node is always thread-safe.

n := db.From("my-node")

Give a bolt.Tx transaction to the Node

n = n.WithTransaction(tx)

Enable batch mode

n = n.WithBatch(true)

Use a Codec

n = n.WithCodec(gob.Codec)

Simple Key/Value store

Storm can be used as a simple, robust, key/value store that can store anything. The key and the value can be of any type as long as the key is not a zero value.

Saving data :

db.Set("logs", time.Now(), "I'm eating my breakfast man")
db.Set("sessions", bson.NewObjectId(), &someUser)
db.Set("weird storage", "754-3010", map[string]interface{}{
  "hair": "blonde",
  "likes": []string{"cheese", "star wars"},
})

Fetching data :

user := User{}
db.Get("sessions", someObjectId, &user)

var details map[string]interface{}
db.Get("weird storage", "754-3010", &details)

db.Get("sessions", someObjectId, &details)

Deleting data :

db.Delete("sessions", someObjectId)
db.Delete("weird storage", "754-3010")

You can find other useful methods in the documentation.

BoltDB

BoltDB is still easily accessible and can be used as usual

db.Bolt.View(func(tx *bolt.Tx) error {
  bucket := tx.Bucket([]byte("my bucket"))
  val := bucket.Get([]byte("any id"))
  fmt.Println(string(val))
  return nil
})

A transaction can be also be passed to Storm

db.Bolt.Update(func(tx *bolt.Tx) error {
  ...
  dbx := db.WithTransaction(tx)
  err = dbx.Save(&user)
  ...
  return nil
})

License

MIT

Credits

Documentation

Index

Examples

Constants

View Source 
const Version = "2.0.0"

Version of Storm

Variables

View Source 
var (
	// ErrNoID is returned when no ID field or id tag is found in the struct.
	ErrNoID = errors.New("missing struct tag id or ID field")

	// ErrZeroID is returned when the ID field is a zero value.
	ErrZeroID = errors.New("id field must not be a zero value")

	// ErrBadType is returned when a method receives an unexpected value type.
	ErrBadType = errors.New("provided data must be a struct or a pointer to struct")

	// ErrAlreadyExists is returned uses when trying to set an existing value on a field that has a unique index.
	ErrAlreadyExists = errors.New("already exists")

	// ErrNilParam is returned when the specified param is expected to be not nil.
	ErrNilParam = errors.New("param must not be nil")

	// ErrUnknownTag is returned when an unexpected tag is specified.
	ErrUnknownTag = errors.New("unknown tag")

	// ErrIdxNotFound is returned when the specified index is not found.
	ErrIdxNotFound = errors.New("index not found")

	// ErrSlicePtrNeeded is returned when an unexpected value is given, instead of a pointer to slice.
	ErrSlicePtrNeeded = errors.New("provided target must be a pointer to slice")

	// ErrStructPtrNeeded is returned when an unexpected value is given, instead of a pointer to struct.
	ErrStructPtrNeeded = errors.New("provided target must be a pointer to struct")

	// ErrPtrNeeded is returned when an unexpected value is given, instead of a pointer.
	ErrPtrNeeded = errors.New("provided target must be a pointer to a valid variable")

	// ErrNoName is returned when the specified struct has no name.
	ErrNoName = errors.New("provided target must have a name")

	// ErrNotFound is returned when the specified record is not saved in the bucket.
	ErrNotFound = errors.New("not found")

	// ErrNotInTransaction is returned when trying to rollback or commit when not in transaction.
	ErrNotInTransaction = errors.New("not in transaction")

	// ErrIncompatibleValue is returned when trying to set a value with a different type than the chosen field
	ErrIncompatibleValue = errors.New("incompatible value")

	// ErrDifferentCodec is returned when using a codec different than the first codec used with the bucket.
	ErrDifferentCodec = errors.New("the selected codec is incompatible with this bucket")

	// ErrUnsupportedSQL is returned when the SQL layer receives syntax it does not execute.
	ErrUnsupportedSQL = errors.New("unsupported sql statement")

	// ErrSQLTableNotRegistered is returned when a SQL table has no registered model.
	ErrSQLTableNotRegistered = errors.New("sql table is not registered")

	// ErrSQLUnknownField is returned when a SQL column cannot be matched to a model field.
	ErrSQLUnknownField = errors.New("sql field is not registered")

	// ErrSQLBadProjectionTarget is returned when Project receives an unsupported destination.
	ErrSQLBadProjectionTarget = errors.New("sql projection target must be a pointer to slice")

	// ErrSQLArguments is returned when SQL placeholders do not match supplied args.
	ErrSQLArguments = errors.New("sql arguments mismatch")

	// ErrSQLUnsafeWrite is returned when UPDATE or DELETE would affect a whole table by default.
	ErrSQLUnsafeWrite = errors.New("sql update/delete without where is not allowed")
)

Errors

Functions

func Batch 

func Batch() func(*Options) error

Batch enables the use of batch instead of update for read-write transactions.

func BleveAsyncWrites added in v2.4.0 

func BleveAsyncWrites() func(*Options) error

BleveAsyncWrites explicitly selects the default asynchronous index behavior: mutating APIs return after the Bolt transaction and durable outbox commit.

func BleveBatchDelay added in v2.4.0 

func BleveBatchDelay(delay time.Duration) func(*Options) error

BleveBatchDelay sets the positive group-commit window used to combine index mutations from concurrent writes after their Bolt transactions commit.

func BleveBatchMaxBytes added in v2.4.0 

func BleveBatchMaxBytes(max uint64) func(*Options) error

BleveBatchMaxBytes sets a positive limit for mapped document bytes in one Bleve batch.

func BleveBatchMaxDocs added in v2.4.0 

func BleveBatchMaxDocs(max int) func(*Options) error

BleveBatchMaxDocs sets a positive limit for document mutations in one Bleve batch.

func BleveBatchQueueSize added in v2.4.0 

func BleveBatchQueueSize(size int) func(*Options) error

BleveBatchQueueSize sets a positive bound for committed write groups waiting for Bleve. Once full, new indexed writes apply backpressure after releasing Bolt's writer lock.

func BleveSyncWrites added in v2.4.0 

func BleveSyncWrites() func(*Options) error

BleveSyncWrites makes mutating APIs wait for their durable outbox entry to be applied to Bleve. It is intended for callers that require synchronous index visibility or immediate Bleve error reporting.

func BoltOptions 

func BoltOptions(mode os.FileMode, options *bolt.Options) func(*Options) error

BoltOptions used to pass options to BoltDB.

func Codec 

func Codec(c codec.MarshalUnmarshaler) func(*Options) error

Codec used to set a custom encoder and decoder. The default is Sonic-backed JSON.

func Debug 

func Debug() func(*Options) error

func Limit 

func Limit(limit int) func(*index.Options)

Limit sets the maximum number of records to return

Example 
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"strings"
	"time"

	storm "github.com/clakeboy/storm-rev/v2"
)

func main() {
	dir, db := prepareDB()
	defer os.RemoveAll(dir)
	defer db.Close()

	var users []User
	err := db.All(&users, storm.Limit(2))

	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Found", len(users))

}

type User struct {
	ID        int    `storm:"id,increment"`
	Group     string `storm:"index"`
	Email     string `storm:"unique"`
	Name      string
	Age       int       `storm:"index"`
	CreatedAt time.Time `storm:"index"`
}

type Account struct {
	ID     int `storm:"id,increment"`
	Amount int64
}

func prepareDB() (string, *storm.DB) {
	dir, _ := ioutil.TempDir(os.TempDir(), "storm")
	db, _ := storm.Open(filepath.Join(dir, "storm.db"))

	for i, name := range []string{"John", "Eric", "Dilbert"} {
		email := strings.ToLower(name + "@provider.com")
		user := User{
			Group:     "staff",
			Email:     email,
			Name:      name,
			Age:       21 + i,
			CreatedAt: time.Now(),
		}
		err := db.Save(&user)

		if err != nil {
			log.Fatal(err)
		}
	}

	for i := int64(0); i < 10; i++ {
		account := Account{Amount: 10000}

		err := db.Save(&account)

		if err != nil {
			log.Fatal(err)
		}
	}

	return dir, db
}

Output:
Found 2

func Reverse 

func Reverse() func(*index.Options)

Reverse will return the results in descending order

func Root 

func Root(root ...string) func(*Options) error

Root used to set the root bucket. See also the From method.

func Skip 

func Skip(offset int) func(*index.Options)

Skip sets the number of records to skip

Example 
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"strings"
	"time"

	storm "github.com/clakeboy/storm-rev/v2"
)

func main() {
	dir, db := prepareDB()
	defer os.RemoveAll(dir)
	defer db.Close()

	var users []User
	err := db.All(&users, storm.Skip(1))

	if err != nil {
		log.Fatal(err)
	}

	fmt.Println("Found", len(users))

}

type User struct {
	ID        int    `storm:"id,increment"`
	Group     string `storm:"index"`
	Email     string `storm:"unique"`
	Name      string
	Age       int       `storm:"index"`
	CreatedAt time.Time `storm:"index"`
}

type Account struct {
	ID     int `storm:"id,increment"`
	Amount int64
}

func prepareDB() (string, *storm.DB) {
	dir, _ := ioutil.TempDir(os.TempDir(), "storm")
	db, _ := storm.Open(filepath.Join(dir, "storm.db"))

	for i, name := range []string{"John", "Eric", "Dilbert"} {
		email := strings.ToLower(name + "@provider.com")
		user := User{
			Group:     "staff",
			Email:     email,
			Name:      name,
			Age:       21 + i,
			CreatedAt: time.Now(),
		}
		err := db.Save(&user)

		if err != nil {
			log.Fatal(err)
		}
	}

	for i := int64(0); i < 10; i++ {
		account := Account{Amount: 10000}

		err := db.Save(&account)

		if err != nil {
			log.Fatal(err)
		}
	}

	return dir, db
}

Output:
Found 2

func UseDB 

func UseDB(b *bolt.DB) func(*Options) error

UseDB allows Storm to use an existing open Bolt.DB. Warning: storm.DB.Close() will close the bolt.DB instance.

Example 
package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"path/filepath"
	"time"

	storm "github.com/clakeboy/storm-rev/v2"

	bolt "go.etcd.io/bbolt"
)

func main() {
	dir, _ := ioutil.TempDir(os.TempDir(), "storm")
	defer os.RemoveAll(dir)

	bDB, err := bolt.Open(filepath.Join(dir, "bolt.db"), 0600, &bolt.Options{Timeout: 10 * time.Second})
	if err != nil {
		log.Fatal(err)
	}

	db, _ := storm.Open("", storm.UseDB(bDB))
	defer db.Close()

	err = db.Save(&User{ID: 10})
	if err != nil {
		log.Fatal(err)
	}

	var user User
	err = db.One("ID", 10, &user)
	fmt.Println(err)

}

type User struct {
	ID        int    `storm:"id,increment"`
	Group     string `storm:"index"`
	Email     string `storm:"unique"`
	Name      string
	Age       int       `storm:"index"`
	CreatedAt time.Time `storm:"index"`
}

Output:
<nil>

Types

type BucketScanner 

type BucketScanner interface {
	// PrefixScan scans the root buckets for keys matching the given prefix.
	PrefixScan(prefix string) []Node
	// PrefixScan scans the buckets in this node for keys matching the given prefix.
	RangeScan(min, max string) []Node
}

A BucketScanner scans a Node for a list of buckets

type ColumnInfo 

type ColumnInfo struct {
	// Name is the Go struct field name.
	Name string `json:"name"`

	// JSON is the JSON field name used when the record is encoded.
	JSON string `json:"json"`

	// Type is the Go type name stored in the table schema.
	Type string `json:"type"`

	// Index is the Storm index tag value, such as index or unique.
	Index string `json:"index,omitempty"`

	// ID reports whether this column is the table ID column.
	ID bool `json:"id,omitempty"`

	// Increment reports whether Storm auto-increments this column.
	Increment bool `json:"increment,omitempty"`

	// IncrementStart is the configured starting value for an increment column.
	IncrementStart int64 `json:"increment_start,omitempty"`

	// Integer reports whether the column stores an integer type.
	Integer bool `json:"integer,omitempty"`

	// Composites records composite-index names and field order.
	Composites map[string]int `json:"composites,omitempty"`
}

ColumnInfo describes one persisted column in a Storm table schema.

type DB 

type DB struct {
	// The root node that points to the root bucket.
	Node

	// Bolt is still easily accessible
	Bolt *bolt.DB
	// contains filtered or unexported fields
}

DB is the wrapper around BoltDB. It contains an instance of BoltDB and uses it to perform all the needed operations

func Open 

func Open(path string, stormOptions ...func(*Options) error) (*DB, error)

Open opens a database at the given path with optional Storm options.

func (*DB) Close 

func (s *DB) Close() error

Close the database

func (*DB) FlushBleve added in v2.4.0 

func (s *DB) FlushBleve(ctx context.Context) error

func (*DB) ListColumns 

func (s *DB) ListColumns(from, table string) ([]ColumnInfo, error)

ListColumns returns persisted column metadata for a table below the given From bucket.

func (*DB) ListFroms 

func (s *DB) ListFroms() ([]string, error)

ListFroms returns direct From bucket names under the current DB root.

func (*DB) ListTables 

func (s *DB) ListTables(from string) ([]string, error)

ListTables returns direct table bucket names below the given From bucket.

func (*DB) SetTableMetadata 

func (s *DB) SetTableMetadata(from, table string, columns []ColumnInfo) error

SetTableMetadata replaces the persisted column metadata for an existing table.

The table bucket must already exist, but it does not need to have __storm_metadata/schema yet. External indexes are not rebuilt here; call ReIndex after changing indexed fields when the new metadata should take effect.

type Finder 

type Finder interface {
	// One returns one record by the specified index
	One(fieldName string, value any, to any) error

	// Find returns one or more records by the specified index
	Find(fieldName string, value any, to any, options ...func(q *index.Options)) error

	// FindByIndex returns records by a named composite index and full equality values.
	FindByIndex(indexName string, values []any, to any, options ...func(q *index.Options)) error

	// Search returns records by a full-text indexed field.
	Search(fieldName string, text string, to any, options ...func(q *index.Options)) error

	// AllByIndex gets all the records of a bucket that are indexed in the specified index
	AllByIndex(fieldName string, to any, options ...func(*index.Options)) error

	// All gets all the records of a bucket.
	// If there are no records it returns no error and the 'to' parameter is set to an empty slice.
	All(to any, options ...func(*index.Options)) error

	// Select a list of records that match a list of matchers. Doesn't use indexes.
	Select(matchers ...q.Matcher) Query

	// Range returns one or more records by the specified index within the specified range
	Range(fieldName string, min, max, to any, options ...func(*index.Options)) error

	// Prefix returns one or more records whose given field starts with the specified prefix.
	Prefix(fieldName string, prefix string, to any, options ...func(*index.Options)) error

	// Count counts all the records of a bucket
	Count(data any) (int, error)
}

A Finder can fetch types from BoltDB.

type KeyValueStore 

type KeyValueStore interface {
	// Get a value from a bucket
	Get(bucketName string, key any, to any) error
	// Set a key/value pair into a bucket
	Set(bucketName string, key any, value any) error
	// Delete deletes a key from a bucket
	Delete(bucketName string, key any) error
	// GetBytes gets a raw value from a bucket.
	GetBytes(bucketName string, key any) ([]byte, error)
	// SetBytes sets a raw value into a bucket.
	SetBytes(bucketName string, key any, value []byte) error
	// KeyExists reports the presence of a key in a bucket.
	KeyExists(bucketName string, key any) (bool, error)
}

KeyValueStore can store and fetch values by key

type Node 

type Node interface {
	Tx
	TypeStore
	KeyValueStore
	BucketScanner

	// From returns a new Storm node with a new bucket root below the current.
	// All DB operations on the new node will be executed relative to this bucket.
	From(addend ...string) Node

	// Bucket returns the bucket name as a slice from the root.
	// In the normal, simple case this will be empty.
	Bucket() []string

	// GetBucket returns the given bucket below the current node.
	GetBucket(tx *bolt.Tx, children ...string) *bolt.Bucket

	// CreateBucketIfNotExists creates the bucket below the current node if it doesn't
	// already exist.
	CreateBucketIfNotExists(tx *bolt.Tx, bucket string) (*bolt.Bucket, error)

	// WithTransaction returns a New Storm node that will use the given transaction.
	WithTransaction(tx *bolt.Tx) Node

	// Begin starts a new transaction.
	Begin(writable bool) (Node, error)

	// Codec used by this instance of Storm
	Codec() codec.MarshalUnmarshaler

	// WithCodec returns a New Storm Node that will use the given Codec.
	WithCodec(codec codec.MarshalUnmarshaler) Node

	// WithBatch returns a new Storm Node with the batch mode enabled.
	WithBatch(enabled bool) Node

	// SQL returns a SQL interpreter bound to this node and the provided models.
	SQL(models ...any) (*SQL, error)
}

A Node in Storm represents the API to a BoltDB bucket.

type Options 

type Options struct {
	// contains filtered or unexported fields
}

Options are used to customize the way Storm opens a database.

type Query 

type Query interface {
	// Skip matching records by the given number
	Skip(int) Query

	// Limit the results by the given number
	Limit(int) Query

	// Order by the given fields, in descending precedence, left-to-right.
	OrderBy(...string) Query

	// Reverse the order of the results
	Reverse() Query

	// Bucket specifies the bucket name
	Bucket(string) Query

	// Find a list of matching records
	Find(any) error

	// First gets the first matching record
	First(any) error

	// Delete all matching records
	Delete(any) error

	// Count all the matching records
	Count(any) (int, error)

	// Returns all the records without decoding them
	Raw() ([][]byte, error)

	// Execute the given function for each raw element
	RawEach(func([]byte, []byte) error) error

	// Execute the given function for each element
	Each(any, func(any) error) error
}

Query is the low level query engine used by Storm. It allows to operate searches through an entire bucket.

type SQL 

type SQL struct {
	// contains filtered or unexported fields
}

SQL interprets a small single-table SQL subset on top of Storm's typed APIs.

func (*SQL) Count 

func (s *SQL) Count(query string, args ...any) (int, error)

Count executes a SELECT COUNT(*) query and returns the matched record count.

func (*SQL) Exec 

func (s *SQL) Exec(query string, args ...any) (SQLResult, error)

Exec executes INSERT, UPDATE, or DELETE and returns affected row metadata.

func (*SQL) Find 

func (s *SQL) Find(query string, to any, args ...any) error

Find executes a SELECT * query and decodes full model records into to.

func (*SQL) First 

func (s *SQL) First(query string, to any, args ...any) error

First executes a SELECT * query and decodes the first full model record into to.

func (*SQL) Project 

func (s *SQL) Project(query string, to any, args ...any) error

Project executes a SELECT query and fills either []map[string]any or a DTO slice.

func (*SQL) Register 

func (s *SQL) Register(models ...any) error

Register adds model metadata used to bind SQL table and column names.

func (*SQL) WithAllowFullTableWrite 

func (s *SQL) WithAllowFullTableWrite(allow bool) *SQL

WithAllowFullTableWrite returns a copy that allows UPDATE/DELETE without WHERE.

type SQLResult 

type SQLResult struct {
	RowsAffected int
	LastInsertID any
}

SQLResult describes the effect of a SQL write statement.

type Tx 

type Tx interface {
	// Commit writes all changes to disk.
	Commit() error

	// Rollback closes the transaction and ignores all previous updates.
	Rollback() error
}

Tx is a transaction.

type TypeStore 

type TypeStore interface {
	Finder
	// Init creates the indexes and buckets for a given structure
	Init(data any) error

	// ReIndex rebuilds all the indexes of a bucket
	ReIndex(data any) error

	// Save a structure
	Save(data any) error

	// SaveAll saves a slice of structures in one write transaction.
	SaveAll(data any) error

	// Update a structure
	Update(data any) error

	// UpdateField updates a single field
	UpdateField(data any, fieldName string, value any) error

	// Drop a bucket
	Drop(data any) error

	// DeleteStruct deletes a structure from the associated bucket
	DeleteStruct(data any) error
}

TypeStore stores user defined types in BoltDB.

Source Files

View all Source files

Directories

Path Synopsis
Package codec contains sub-packages with different codecs that can be used to encode and decode entities in Storm.
 Package codec contains sub-packages with different codecs that can be used to encode and decode entities in Storm.
aes
gob
Package gob contains a codec to encode and decode entities in Gob format
 Package gob contains a codec to encode and decode entities in Gob format
internal
json
Package json contains a codec to encode and decode entities in JSON format
 Package json contains a codec to encode and decode entities in JSON format
msgpack
Package msgpack contains a codec to encode and decode entities in msgpack format
 Package msgpack contains a codec to encode and decode entities in msgpack format
protobuf
Package protobuf contains a codec to encode and decode entities in Protocol Buffer
 Package protobuf contains a codec to encode and decode entities in Protocol Buffer
sereal
Package sereal contains a codec to encode and decode entities using Sereal
 Package sereal contains a codec to encode and decode entities using Sereal
Package index contains Index engines used to store values and their corresponding IDs
 Package index contains Index engines used to store values and their corresponding IDs
q
Package q contains a list of Matchers used to compare struct fields with values
 Package q contains a list of Matchers used to compare struct fields with values

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.