clover

README

Lightweight document-oriented NoSQL Database

🇬🇧 English | 🇨🇳 简体中文 | 🇪🇸 Spanish

CloverDB is a lightweight NoSQL database designed for being simple and easily maintainable, thanks to its small code base. It has been inspired by tinyDB.

Features

• Document oriented
• Written in pure Golang
• Simple and intuitive api
• Easily maintainable

Why CloverDB?

CloverDB has been written for being easily maintenable. As such, it trades performance with simplicity, and is not intented to be an alternative to more performant databases such as MongoDB or MySQL. However, there are projects where running a separate database server may result overkilled, and, for simple queries, network delay may be the major performance bottleneck. For there scenario, CloverDB may be a more suitable alternative.

Database Layout

CloverDB abstracts the way collections are stored on disk through the StorageEngine interface. The default implementation is based on the Badger database key-value store. However, you could easily write your own storage engine implementation.

Installation

Make sure you have a working Go environment (Go 1.13 or higher is required).

  GO111MODULE=on go get github.com/ostafen/clover

Databases and Collections

CloverDB stores data records as JSON documents, which are grouped together in collections. A database is made up of one or more collections.

Database

To store documents inside collections, you have to open a Clover database using the Open() function.

import (
	"log"
	c "github.com/ostafen/clover"
)

...

db, _ := c.Open("clover-db")

// or, if you don't need persistency
db, _ := c.Open("", c.InMemoryMode(true))

defer db.Close() // remember to close the db when you have done

Collections

CloverDB stores documents inside collections. Collections are the schemaless equivalent of tables in relational databases. A collection is created by calling the CreateCollection() function on a database instance. New documents can be inserted using the Insert() or InsertOne() methods. Each document is uniquely identified by a Version 4 UUID stored in the _id special field and generated during insertion.

db, _ := c.Open("clover-db")
db.CreateCollection("myCollection") // create a new collection named "myCollection"

// insert a new document inside the collection
doc := c.NewDocument()
doc.Set("hello", "clover!")

// InsertOne returns the id of the inserted document
docId, _ := db.InsertOne("myCollection", doc)
fmt.Println(docId)

Importing and Exporting Collections

CloverDB is capable of easily importing and exporting collections to JSON format regardless of the storage engine used.

// dump the content of the "todos" collection in a "todos.json" file
db.ExportCollection("todos", "todos.json")

...

// recover the todos collection from the exported json file
db.DropCollection("todos")
db.ImportCollection("todos", "todos.json")

docs, _ := db.Query("todos").FindAll()
for _, doc := range docs {
  log.Println(doc)
}

Queries

CloverDB is equipped with a fluent and elegant API to query your data. A query is represented by the Query object, which allows to retrieve documents matching a given criterion. A query can be created by passing a valid collection name to the Query() method.

Select All Documents in a Collection

The FindAll() method is used to retrieve all documents satisfying a given query.

docs, _ := db.Query("myCollection").FindAll()

todo := &struct {
    Completed bool   `clover:"completed"`
    Title     string `clover:"title"`
    UserId    int    `clover:"userId"`
}{}

for _, doc := range docs {
    doc.Unmarshal(todo)
    log.Println(todo)
}

Filter Documents with Criteria

In order to filter the documents returned by FindAll(), you have to specify a query Criteria using the Where() method. A Criteria object simply represents a predicate on a document, evaluating to true only if the document satisfies all the query conditions.

The following example shows how to build a simple Criteria, matching all the documents having the completed field equal to true.

db.Query("todos").Where(c.Field("completed").Eq(true)).FindAll()

// or equivalently
db.Query("todos").Where(c.Field("completed").IsTrue()).FindAll()

In order to build very complex queries, we chain multiple Criteria objects by using the And() and Or() methods, each returning a new Criteria obtained by appling the corresponding logical operator.

// find all completed todos belonging to users with id 5 and 8
db.Query("todos").Where(c.Field("completed").Eq(true).And(c.Field("userId").In(5, 8))).FindAll()

Sorting Documents

To sort documents in CloverDB, you need to use Sort(). It is a variadic function which accepts a sequence of SortOption, each allowing to specify a field and a sorting direction. A sorting direction can be one of 1 or -1, respectively corresponding to ascending and descending order. If no SortOption is provided, Sort() uses the _id field by default.

// Find any todo belonging to the most recent inserted user
db.Query("todos").Sort(c.SortOption{"userId", -1}).FindFirst()

Skip/Limit Documents

Sometimes, it can be useful to discard some documents from the output, or simply set a limit on the maximum number of results returned by a query. For this purpose, CloverDB provides the Skip() and Limit() functions, both accepting an interger $n$ as parameter.

// discard the first 10 documents from the output,
// also limiting the maximum number of query results to 100
db.Query("todos").Skip(10).Limit(100).FindAll()

Update/Delete Documents

The Update() method is used to modify specific fields of documents in a collection. The Delete() method is used to delete documents. Both methods belong to the Query object, so that it is easy to update and delete documents matching a particular query.

// mark all todos belonging to user with id 1 as completed
updates := make(map[string]interface{})
updates["completed"] = true

db.Query("todos").Where(c.Field("userId").Eq(1)).Update(updates)

// delete all todos belonging to users with id 5 and 8
db.Query("todos").Where(c.Field("userId").In(5,8)).Delete()

To update or delete a single document using a specific document id, use UpdateById() or DeleteById(), respectively:

docId := "1dbce353-d3c6-43b3-b5a8-80d8d876389b"
// update the document with the specified id
db.Query("todos").UpdateById(docId, map[string]interface{}{"completed": true})
// or delete it
db.Query("todos").DeleteById(docId)

Data Types

Internally, CloverDB supports the following primitive data types: int64, uint64, float64, string, bool and time.Time. When possible, values having different types are silently converted to one of the internal types: signed integer values get converted to int64, while unsigned ones to uint64. Float32 values are extended to float64.

For example, consider the following snippet, which sets an uint8 value on a given document field:

doc := c.NewDocument()
doc.Set("myField", uint8(10)) // "myField" is automatically promoted to uint64

fmt.Println(doc.Get("myField").(uint64))

Pointer values are dereferenced until either nil or a non-pointer value is found:

var x int = 10
var ptr *int = &x
var ptr1 **int = &ptr

doc.Set("ptr", ptr)
doc.Set("ptr1", ptr1)

fmt.Println(doc.Get("ptr").(int64) == 10)
fmt.Println(doc.Get("ptr1").(int64) == 10)

ptr = nil

doc.Set("ptr1", ptr1)
// ptr1 is not nil, but it points to the nil "ptr" pointer, so the field is set to nil
fmt.Println(doc.Get("ptr1") == nil)

Invalid types leaves the document untouched:

doc := c.NewDocument()
doc.Set("myField", make(chan struct{}))

log.Println(doc.Has("myField")) // will output false

Contributing

CloverDB is actively developed. Any contribution, in the form of a suggestion, bug report or pull request, is well accepted 😊

Major contributions and suggestions have been gratefully received from (in alphabetical order):

• ASWLaunchs
• jsgm
• segfault99

Documentation

Index

• Variables
• func Field(name string) *field
• func NewObjectId() string
• type Config
• type Criteria
  • func (c *Criteria) And(other *Criteria) *Criteria
  • func (c *Criteria) Not() *Criteria
  • func (c *Criteria) Or(other *Criteria) *Criteria
• type DB
  • func Open(dir string, opts ...Option) (*DB, error)
  • func (db *DB) Close() error
  • func (db *DB) CreateCollection(name string) error
  • func (db *DB) DropCollection(name string) error
  • func (db *DB) ExportCollection(collectionName string, exportPath string) error
  • func (db *DB) HasCollection(name string) (bool, error)
  • func (db *DB) ImportCollection(collectionName string, importPath string) error
  • func (db *DB) Insert(collectionName string, docs ...*Document) error
  • func (db *DB) InsertOne(collectionName string, doc *Document) (string, error)
  • func (db *DB) ListCollections() ([]string, error)
  • func (db *DB) Query(name string) *Query
  • func (db *DB) Save(collectionName string, doc *Document) error
• type Document
  • func NewDocument() *Document
  • func NewDocumentOf(o interface{}) *Document
  • func (doc *Document) Copy() *Document
  • func (doc *Document) Get(name string) interface{}
  • func (doc *Document) Has(name string) bool
  • func (doc *Document) ObjectId() string
  • func (doc *Document) Set(name string, value interface{})
  • func (doc *Document) SetAll(values map[string]interface{})
  • func (doc *Document) Unmarshal(v interface{}) error
• type Option
  • func InMemoryMode(enable bool) Option
  • func WithStorageEngine(engine StorageEngine) Option
• type Query
  • func (q *Query) Count() (int, error)
  • func (q *Query) Delete() error
  • func (q *Query) DeleteById(id string) error
  • func (q *Query) Exists() (bool, error)
  • func (q *Query) FindAll() ([]*Document, error)
  • func (q *Query) FindById(id string) (*Document, error)
  • func (q *Query) FindFirst() (*Document, error)
  • func (q *Query) ForEach(consumer func(_ *Document) bool) error
  • func (q *Query) Limit(n int) *Query
  • func (q *Query) MatchPredicate(p func(doc *Document) bool) *Query
  • func (q *Query) ReplaceById(docId string, doc *Document) error
  • func (q *Query) Skip(n int) *Query
  • func (q *Query) Sort(opts ...SortOption) *Query
  • func (q *Query) Update(updateMap map[string]interface{}) error
  • func (q *Query) UpdateById(docId string, updateMap map[string]interface{}) error
  • func (q *Query) Where(c *Criteria) *Query
• type SortOption
• type StorageEngine

Variables

var (
	ErrCollectionExist    = errors.New("collection already exist")
	ErrCollectionNotExist = errors.New("no such collection")
)

Collection creation errors

var ErrDocumentNotExist = errors.New("no such document")

var ErrDuplicateKey = errors.New("duplicate key")

Functions

func Field

func Field(name string) *field

Field represents a document field. It is used to create a new criteria.

func NewObjectId

func NewObjectId() string

Types

type Config

type Config struct {
	InMemory bool
	Storage  StorageEngine
}

Config contains clover configuration parameters

type Criteria

type Criteria struct {
	// contains filtered or unexported fields
}

Criteria represents a predicate for selecting documents. It follows a fluent API style so that you can easily chain together multiple criteria.

func (*Criteria) And

func (c *Criteria) And(other *Criteria) *Criteria

And returns a new Criteria obtained by combining the predicates of the provided criteria with the AND logical operator.

func (*Criteria) Not

func (c *Criteria) Not() *Criteria

Not returns a new Criteria which negate the predicate of the original criterion.

func (*Criteria) Or

func (c *Criteria) Or(other *Criteria) *Criteria

Or returns a new Criteria obtained by combining the predicates of the provided criteria with the OR logical operator.

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB represents the entry point of each clover database.

func Open

func Open(dir string, opts ...Option) (*DB, error)

Open opens a new clover database on the supplied path. If such a folder doesn't exist, it is automatically created.

func (*DB) Close

func (db *DB) Close() error

Close releases all the resources and closes the database. After the call, the instance will no more be usable.

func (*DB) CreateCollection

func (db *DB) CreateCollection(name string) error

CreateCollection creates a new empty collection with the given name.

func (*DB) DropCollection

func (db *DB) DropCollection(name string) error

DropCollection removes the collection with the given name, deleting any content on disk.

func (*DB) ExportCollection

func (db *DB) ExportCollection(collectionName string, exportPath string) error

ExportCollection exports an existing collection to a JSON file.

func (*DB) HasCollection

func (db *DB) HasCollection(name string) (bool, error)

HasCollection returns true if and only if the database contains a collection with the given name.

func (*DB) ImportCollection

func (db *DB) ImportCollection(collectionName string, importPath string) error

ImportCollection imports a collection from a JSON file.

func (*DB) Insert

func (db *DB) Insert(collectionName string, docs ...*Document) error

Insert adds the supplied documents to a collection.

func (*DB) InsertOne

func (db *DB) InsertOne(collectionName string, doc *Document) (string, error)

InsertOne inserts a single document to an existing collection. It returns the id of the inserted document.

func (*DB) ListCollections

func (db *DB) ListCollections() ([]string, error)

ListCollections returns a slice of strings containing the name of each collection stored in the db.

func (*DB) Query

func (db *DB) Query(name string) *Query

Query simply returns the collection with the supplied name. Use it to initialize a new query.

func (*DB) Save

func (db *DB) Save(collectionName string, doc *Document) error

Save or update a document

type Document

type Document struct {
	// contains filtered or unexported fields
}

Document represents a document as a map.

func NewDocument

func NewDocument() *Document

NewDocument creates a new empty document.

func NewDocumentOf

func NewDocumentOf(o interface{}) *Document

NewDocumentOf creates a new document and initializes it with the content of the provided object. It returns nil if the object cannot be converted to a valid Document.

func (*Document) Copy

func (doc *Document) Copy() *Document

Copy returns a shallow copy of the underlying document.

func (*Document) Get

func (doc *Document) Get(name string) interface{}

Get retrieves the value of a field. Nested fields can be accessed using dot.

func (*Document) Has

func (doc *Document) Has(name string) bool

Has tells returns true if the document contains a field with the supplied name.

func (*Document) ObjectId

func (doc *Document) ObjectId() string

ObjectId returns the id of the document, provided that the document belongs to some collection. Otherwise, it returns the empty string.

func (*Document) Set

func (doc *Document) Set(name string, value interface{})

Set maps a field to a value. Nested fields can be accessed using dot.

func (*Document) SetAll

func (doc *Document) SetAll(values map[string]interface{})

SetAll sets each field specified in the input map to the corresponding value. Nested fields can be accessed using dot.

func (*Document) Unmarshal

func (doc *Document) Unmarshal(v interface{}) error

Unmarshal stores the document in the value pointed by v.

type Option

type Option func(c *Config) error

Option is a function that takes a config struct and modifies it

func InMemoryMode

func InMemoryMode(enable bool) Option

InMemoryMode allows to enable/disable in-memory mode.

func WithStorageEngine

func WithStorageEngine(engine StorageEngine) Option

WithStorageEngine allows to specify a custom storage engine.

type Query

type Query struct {
	// contains filtered or unexported fields
}

Query represents a generic query which is submitted to a specific collection.

func (*Query) Count

func (q *Query) Count() (int, error)

Count returns the number of documents which satisfy the query (i.e. len(q.FindAll()) == q.Count()).

func (*Query) Delete

func (q *Query) Delete() error

Delete removes all the documents selected by q from the underlying collection.

func (*Query) DeleteById

func (q *Query) DeleteById(id string) error

DeleteById removes the document with the given id from the underlying collection, provided that such a document exists and satisfies the underlying query.

func (*Query) Exists

func (q *Query) Exists() (bool, error)

Exists returns true if and only if the query result set is not empty.

func (*Query) FindAll

func (q *Query) FindAll() ([]*Document, error)

FindAll selects all the documents satisfying q.

func (*Query) FindById

func (q *Query) FindById(id string) (*Document, error)

FindById returns the document with the given id, if such a document exists and satisfies the underlying query, or null.

func (*Query) FindFirst

func (q *Query) FindFirst() (*Document, error)

FindFirst returns the first document (if any) satisfying the query.

func (*Query) ForEach

func (q *Query) ForEach(consumer func(_ *Document) bool) error

ForEach runs the consumer function for each document matching the provied query. If false is returned from the consumer function, then the iteration is stopped.

func (*Query) Limit

func (q *Query) Limit(n int) *Query

Limit sets the query q to consider at most n records. As a consequence, the FindAll() method will output at most n documents, and any integer m returned by Count() will satisfy the condition m <= n.

func (*Query) MatchPredicate

func (q *Query) MatchPredicate(p func(doc *Document) bool) *Query

MatchPredicate selects all the documents which satisfy the supplied predicate function.

func (*Query) ReplaceById

func (q *Query) ReplaceById(docId string, doc *Document) error

ReplaceById replaces the document with the specified id with the one provided. If no document exists, an ErrDocumentNotExist is returned.

func (*Query) Skip

func (q *Query) Skip(n int) *Query

func (*Query) Sort

func (q *Query) Sort(opts ...SortOption) *Query

Sort sets the query so that the returned documents are sorted according list of options.

func (*Query) Update

func (q *Query) Update(updateMap map[string]interface{}) error

Update updates all the document selected by q using the provided updateMap. Each update is specified by a mapping fieldName -> newValue.

func (*Query) UpdateById

func (q *Query) UpdateById(docId string, updateMap map[string]interface{}) error

UpdateById updates the document with the specified id using the supplied update map. If no document with the specified id exists, an ErrDocumentNotExist is returned.

func (*Query) Where

func (q *Query) Where(c *Criteria) *Query

Where returns a new Query which select all the documents fullfilling both the base query and the provided Criteria.

type SortOption

type SortOption struct {
	Field     string
	Direction int
}

SortOption is used to specify sorting options to the Sort method. It consists of a field name and a sorting direction (1 for ascending and -1 for descending). Any other positive of negative value (except from 1 and -1) will be equivalent, respectively, to 1 or -1. A direction value of 0 (which is also the default value) is assumed to be ascending.

type StorageEngine

type StorageEngine interface {
	Open(path string) error
	Close() error

	CreateCollection(name string) error
	ListCollections() ([]string, error)
	DropCollection(name string) error
	HasCollection(name string) (bool, error)
	FindAll(q *Query) ([]*Document, error)
	FindById(collectionName string, id string) (*Document, error)
	UpdateById(collectionName string, docId string, updater func(doc *Document) *Document) error
	DeleteById(collectionName string, id string) error
	IterateDocs(q *Query, consumer docConsumer) error
	Insert(collection string, docs ...*Document) error
	Update(q *Query, updater func(doc *Document) *Document) error
	Delete(q *Query) error
}

StorageEngine represents the persistance layer and abstracts how collections are stored.