unitdb

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2020 License: Apache-2.0 Imports: 29 Imported by: 0

README

unitdb GoDoc Go Report Card Build Status Coverage Status

Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. The unitdb satisfy the requirements for low latency and binary messaging, it is a perfect time-series database for applications such as internet of things and internet connected devices.

About unitdb

Key characteristics

  • 100% Go
  • Optimized for fast lookups and writes
  • Can store larger-than-memory data sets
  • Data is safely written to disk with accuracy and high performant block sync technique
  • Supports opening database with immutable flag
  • Supports database encryption
  • Supports time-to-live on message entry
  • Supports writing to wildcard topics
  • Queried data is returned complete and correct

Quick Start

To build unitdb from source code use go get command.

go get -u github.com/unit-io/unitdb

Usage

Detailed API documentation is available using the go.dev service.

Make use of the client by importing it in your Go client source code. For example,

import "github.com/unit-io/unitdb"

The unitdb supports Get, Put, Delete operations. It also supports encryption, batch operations, and writing to wildcard topics. See usage guide.

Samples are available in the cmd directory for reference.

Projects Using Unitdb

Below is a list of projects that use unitdb.

  • Unitd Lightweight, high performance messaging system for microservices, IoT, and internet connected devices.
  • Unitd Go Client Lightweight and high performance publish-subscribe messaging system - Go client library.

Contributing

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.

Licensing

This project is licensed under Apache-2.0 License.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Debug 

func Debug(context, msg string)

Debug logs the debug message with tag if it is turned on.

func Fatal 

func Fatal(context, msg string, err error)

Fatal logs the fatal error messages.

func Info 

func Info(context, action string)

Info logs the action with a tag.

func ParseLevel 

func ParseLevel(level string, defaultLevel zerolog.Level) zerolog.Level

ParseLevel parses a string which represents a log level and returns a zerolog.Level.

func ResponseHandler 

func ResponseHandler(w http.ResponseWriter, r *http.Request, data []byte)

ResponseHandler handles responses for monitoring routes.

Types

type Batch 

type Batch struct {
	// contains filtered or unexported fields
}

Batch is a write batch.

func (*Batch) Abort 

func (b *Batch) Abort()

Abort abort is a batch cleanup operation on batch complete.

func (*Batch) Commit 

func (b *Batch) Commit() error

Commit commits changes to the DB. In batch operation commit is managed and client is not allowed to call Commit. On Commit complete batch operation signal to the caller if the batch is fully commited to DB.

func (*Batch) Delete 

func (b *Batch) Delete(id, topic []byte) error

Delete appends delete entry to batch for given key. It is safe to modify the contents of the argument after Delete returns but not before.

func (*Batch) DeleteEntry 

func (b *Batch) DeleteEntry(e *Entry) error

DeleteEntry appends entry for deletion to a batch for given key. It is safe to modify the contents of the argument after Delete returns but not before.

func (*Batch) Put 

func (b *Batch) Put(topic, payload []byte) error

Put adds entry to batch for given topic->key/value. Client must provide Topic to the BatchOptions. It is safe to modify the contents of the argument after Put returns but not before.

func (*Batch) PutEntry 

func (b *Batch) PutEntry(e *Entry) error

PutEntry appends entries to a bacth for given topic->key/value pair. It is safe to modify the contents of the argument after Put returns but not before.

func (*Batch) SetOptions 

func (b *Batch) SetOptions(opts ...Options)

SetOptions sets batch options.

func (*Batch) Write 

func (b *Batch) Write() error

Write starts writing entries into DB. It returns an error if batch write fails.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents the message storage for topic->keys-values. All DB methods are safe for concurrent use by multiple goroutines.

func Open 

func Open(path string, opts ...Options) (*DB, error)

Open opens or creates a new DB.

func (*DB) Batch 

func (db *DB) Batch(fn func(*Batch, <-chan struct{}) error) error

Batch executes a function within the context of a read-write managed transaction. If no error is returned from the function then the transaction is written. If an error is returned then the entire transaction is rolled back. Any error that is returned from the function or returned from the write is returned from the Batch() method.

Attempting to manually commit or rollback within the function will cause a panic.

func (*DB) Close 

func (db *DB) Close() error

Close closes the DB.

func (*DB) Count 

func (db *DB) Count() uint64

Count returns the number of items in the DB.

func (*DB) Delete 

func (db *DB) Delete(id, topic []byte) error

Delete sets entry for deletion. It is safe to modify the contents of the argument after Delete returns but not before.

func (*DB) DeleteEntry 

func (db *DB) DeleteEntry(e *Entry) error

DeleteEntry deletes an entry from DB. you must provide an ID to delete an entry. It is safe to modify the contents of the argument after Delete returns but not before.

func (*DB) FileSize 

func (db *DB) FileSize() (int64, error)

FileSize returns the total size of the disk storage used by the DB.

func (*DB) Get 

func (db *DB) Get(q *Query) (items [][]byte, err error)

Get return items matching the query paramater.

func (*DB) HandleVarz 

func (db *DB) HandleVarz(w http.ResponseWriter, r *http.Request)

HandleVarz will process HTTP requests for unitdb stats information.

func (*DB) Items 

func (db *DB) Items(q *Query) (*ItemIterator, error)

Items returns a new ItemIterator.

func (*DB) NewContract 

func (db *DB) NewContract() (uint32, error)

NewContract generates a new Contract.

func (*DB) NewID 

func (db *DB) NewID() []byte

NewID generates new ID that is later used to put entry or delete entry.

func (*DB) Put 

func (db *DB) Put(topic, payload []byte) error

Put puts entry into DB. It uses default Contract to put entry into DB. It is safe to modify the contents of the argument after Put returns but not before.

func (*DB) PutEntry 

func (db *DB) PutEntry(e *Entry) error

PutEntry puts entry into the DB, if Contract is not specified then it uses master Contract. It is safe to modify the contents of the argument after PutEntry returns but not before.

func (*DB) Sync 

func (db *DB) Sync() error

Sync syncs entries into DB. Sync happens synchronously. Sync write window entries into summary file and write index, and data to respective index and data files. In case of any error during sync operation recovery is performed on log file (write ahead log).

func (*DB) Varz 

func (db *DB) Varz() (*Varz, error)

Varz returns a Varz struct containing the unitdb information.

type Entry 

type Entry struct {
	ID         []byte // The ID of the message.
	Topic      []byte // The topic of the message.
	Payload    []byte // The payload of the message.
	ExpiresAt  uint32 // The time expiry of the message.
	Contract   uint32 // The contract is used to as salt to hash topic parts and also used as prefix in the message ID.
	Encryption bool
	// contains filtered or unexported fields
}

Entry entry is a message entry structure.

func NewEntry 

func NewEntry(topic, payload []byte) *Entry

NewEntry creates a new entry structure from the topic.

func (Entry) ExpiresAt 

func (e Entry) ExpiresAt() uint32

func (Entry) MarshalBinary 

func (e Entry) MarshalBinary() ([]byte, error)

MarshalBinary serialized entry into binary data.

func (*Entry) UnmarshalBinary 

func (e *Entry) UnmarshalBinary(data []byte) error

MarshalBinary de-serialized entry from binary data.

func (*Entry) WithContract 

func (e *Entry) WithContract(contract uint32) *Entry

WithContract sets contract on entry.

func (*Entry) WithEncryption 

func (e *Entry) WithEncryption() *Entry

WithEncryption sets encryption on entry.

func (*Entry) WithID 

func (e *Entry) WithID(id []byte) *Entry

WithID sets entry ID.

func (*Entry) WithPayload 

func (e *Entry) WithPayload(payload []byte) *Entry

WithPayload sets payload to put entry into DB.

func (*Entry) WithTTL 

func (e *Entry) WithTTL(ttl []byte) *Entry

WithTTL sets TTL for message expiry for the entry.

type Filter 

type Filter struct {
	// contains filtered or unexported fields
}

Filter filter is bloom filter generator.

func (*Filter) Append 

func (f *Filter) Append(h uint64)

Append appends an entry to bloom filter.

func (*Filter) Size 

func (f *Filter) Size() int64

func (*Filter) Test 

func (f *Filter) Test(h uint64) bool

Test tests entry in bloom filter. It returns false if entry definitely does not exist or entry maybe existing in DB.

type Item 

type Item struct {
	// contains filtered or unexported fields
}

Item items returned by the iterator.

func (*Item) Topic 

func (item *Item) Topic() []byte

Topic returns the topic of the current item, or nil if done. The caller should not modify the contents of the returned slice, and its contents may change on the next call to Next.

func (*Item) Value 

func (item *Item) Value() []byte

Value returns the value of the current item, or nil if done. The caller should not modify the contents of the returned slice, and its contents may change on the next call to Next.

type ItemIterator 

type ItemIterator struct {
	// contains filtered or unexported fields
}

ItemIterator is an iterator over DB topic->key/value pairs. It iterates the items in an unspecified order.

func (*ItemIterator) Error 

func (it *ItemIterator) Error() error

Error returns any accumulated error. Exhausting all the key/value pairs is not considered to be an error. A memory iterator cannot encounter errors.

func (*ItemIterator) First 

func (it *ItemIterator) First()

First is similar to init. It query and loads window entries from trie/timeWindowBucket or summary file if available.

func (*ItemIterator) Item 

func (it *ItemIterator) Item() *Item

Item returns pointer to the current item. This item is only valid until it.Next() gets called.

func (*ItemIterator) Next 

func (it *ItemIterator) Next()

Next returns the next topic->key/value pair if available, otherwise it returns ErrIterationDone error.

func (*ItemIterator) Release 

func (it *ItemIterator) Release()

Release releases associated resources. Release should always succeed and can be called multiple times without causing error.

func (*ItemIterator) Valid 

func (it *ItemIterator) Valid() bool

Valid returns false when iteration is done.

type Meter 

type Meter struct {
	Metrics    metrics.Metrics
	TimeSeries metrics.TimeSeries
	Gets       metrics.Counter
	Puts       metrics.Counter
	Leased     metrics.Counter
	Syncs      metrics.Counter
	Recovers   metrics.Counter
	Dels       metrics.Counter
	InMsgs     metrics.Counter
	OutMsgs    metrics.Counter
	InBytes    metrics.Counter
	OutBytes   metrics.Counter
}

Meter meter provides various db statistics.

func NewMeter 

func NewMeter() *Meter

NewMeter provide meter to capture statistics.

func (*Meter) UnregisterAll 

func (m *Meter) UnregisterAll()

UnregisterAll unregister all metrics from meter.

type Options 

type Options interface {
	// contains filtered or unexported methods
}

Options it contains configurable options and flags for DB.

func TinyBatchWriteInterval 

func TinyBatchWriteInterval(dur time.Duration) Options

WithTinyBatchWriteInterval sets interval to group tiny batches and write into db on tiny batch interval.

func WithBackgroundKeyExpiry 

func WithBackgroundKeyExpiry() Options

WithBackgroundKeyExpiry sets background key expiry for DB.

func WithBatchContract 

func WithBatchContract(contract uint32) Options

WithBatchContract sets contract for batch operation.

func WithBatchEncryption 

func WithBatchEncryption() Options

WithBatchEncryption sets encryption on batch operation.

func WithBatchWriteInterval 

func WithBatchWriteInterval(dur time.Duration) Options

WithBatchWriteInterval sets batch write interval to partial write large batch.

func WithBufferSize 

func WithBufferSize(size int64) Options

WithBufferSize sets Size of buffer to use for pooling.

func WithDefaultBatchOptions 

func WithDefaultBatchOptions() Options

WithDefaultBatchOptions will set some default values for Batch operation. 

contract: MasterContract
encryption: False

func WithDefaultFlags 

func WithDefaultFlags() Options

WithDefaultFlags will open DB with some default values. 

immutable: True
encryption: False
backgroundKeyExpiry: False

func WithDefaultOptions 

func WithDefaultOptions() Options

WithDefaultOptions will open DB with some default values.

func WithDefaultQueryLimit 

func WithDefaultQueryLimit(limit int) Options

WithDefaultQueryLimit limits maximum number of records to fetch if the DB Get or DB Iterator method does not specify a limit.

func WithEncryption 

func WithEncryption() Options

WithEncryption sets encryption on DB.

func WithEncryptionKey 

func WithEncryptionKey(key []byte) Options

WithEncryptionKey sets encryption key to use for data encryption.

func WithLogSize 

func WithLogSize(size int64) Options

WithLogSize sets Size of write ahead log.

func WithMaxQueryLimit 

func WithMaxQueryLimit(limit int) Options

WithMaxQueryLimit limits maximum number of records to fetch if the DB Get or DB Iterator method does not specify a limit or specify a limit larger than MaxQueryResults.

func WithMaxSyncDuration 

func WithMaxSyncDuration(dur time.Duration, interval int) Options

WithBackgroundSyncInterval sets the amount of time between background fsync() calls.

func WithMemdbSize 

func WithMemdbSize(size int64) Options

WithMemdbSize sets Size of memory DB.

func WithMinimumFreeBlocksSize 

func WithMinimumFreeBlocksSize(size int64) Options

WithMinimumFreeBlocksSize sets minimum freeblocks size before free blocks are allocated and reused.

func WithMutable 

func WithMutable() Options

WithMutable sets Immutable flag to false.

type Query 

type Query struct {
	Topic    []byte // The topic of the message.
	Contract uint32 // The contract is used as prefix in the message ID.
	Limit    int    // The maximum number of elements to return.
	// contains filtered or unexported fields
}

Query represents a topic to query and optional contract information.

func NewQuery 

func NewQuery(topic []byte) *Query

NewQuery creates a new query structure from the topic.

func (*Query) WithContract 

func (q *Query) WithContract(contract uint32) *Query

WithContract sets contract on query.

func (*Query) WithLimit 

func (q *Query) WithLimit(limit int) *Query

WithLimit sets query limit.

type Varz 

type Varz struct {
	Start    time.Time `json:"start"`
	Now      time.Time `json:"now"`
	Uptime   string    `json:"uptime"`
	Seq      int64     `json:"seq"`
	Count    int64     `json:"count"`
	Blocks   int64     `json:"blocks"`
	Gets     int64     `json:"gets"`
	Puts     int64     `json:"puts"`
	Leased   int64     `json:"leased"`
	Syncs    int64     `json:"syncs"`
	Recovers int64     `json:"recovers"`
	Dels     int64     `json:"Dels"`
	InMsgs   int64     `json:"in_msgs"`
	OutMsgs  int64     `json:"out_msgs"`
	InBytes  int64     `json:"in_bytes"`
	OutBytes int64     `json:"out_bytes"`
	HMean    float64   `json:"hmean"` // Event duration harmonic mean.
	P50      float64   `json:"p50"`   // Event duration nth percentiles.
	P75      float64   `json:"p75"`
	P95      float64   `json:"p95"`
	P99      float64   `json:"p99"`
	P999     float64   `json:"p999"`
	Long5p   float64   `json:"long_5p"`  // Average of the longest 5% event durations.
	Short5p  float64   `json:"short_5p"` // Average of the shortest 5% event durations.
	Max      float64   `json:"max"`      // Highest event duration.
	Min      float64   `json:"min"`      // Lowest event duration.
	StdDev   float64   `json:"stddev"`   // Standard deviation.

}

Varz outputs unitdb stats on the monitoring port at /varz.

Source Files

View all Source files

Directories

Path Synopsis
cmd
benchmark
sample
simple
Package hash provides a minimal-memory AnchorHash consistent-hash implementation for Go.
 Package hash provides a minimal-memory AnchorHash consistent-hash implementation for Go.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.