leia

package module
v2.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 4, 2022 License: GPL-3.0 Imports: 17 Imported by: 4

README

Build Coverage Maintainability

go-leia

Go Lightweight Embedded Indexed (JSON) Archive

go-leia is built upon bbolt. It adds indexed based search capabilities for JSON documents to the key-value store.

The goal is to provide a simple and fast way to find relevant JSON documents using an embedded Go key-value store.

Table of Contents

Installing

Install Go and run go get:

$ go get github.com/nuts-foundation/go-leia

When using Go > 1.16, Go modules will probably require you to install additional dependencies.

$ go get github.com/stretchr/testify
$ go get github.com/tidwall/gjson
$ go get go.etcd.io/bbolt

Opening a database

Opening a database only requires a file location for the bbolt db.

package main

import (
	"log"
	
	"github.com/nuts-foundation/go-leia"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created if it doesn't exist using filemode 0600 and default bbolt options.
	store, err := leia.NewStore("my.db")
	if err != nil {
		log.Fatal(err)
	}
	defer store.Close()

	...
}

Collections

Leia adds collections to bbolt. Each collection has its own bucket where documents are stored. An index is also only valid for a single collection.

To create a collection:

func main() {
    store, err := leia.NewStore("my.db")
	...
	
    // if a collection doesn't exist, it'll be created for you.
    // the underlying buckets are created when a document is added.
    collection := store.Collection("credentials")
}
Writing

Writing a document to a collection is straightforward:

func main() {
    store, err := leia.NewStore("my.db")
    collection := store.Collection("credentials")
	...
	
    // leia uses leia.Documents as arguments. Which is basically a []byte
    documents := make([]leia.Document, 1)
    documents[1] = leia.DocumentFromString("{...some json...}")
    
    // documents are added by slice
    collection.Add(documents)
}

Documents are added by slice. Each operation is done within a single bbolt transaction. BBolt is a key-value store, so you've probably noticed the key is missing as an argument. Leia computes the sha-1 of the document and uses that as key.

To get the key when needed:

func main() {
    store, err := leia.NewStore("my.db")
    collection := store.Collection("credentials")
    ...
    
    // define your document
    document := leia.DocumentFromString("{...some json...}")
    
    // retrieve a leia.Reference (also a []byte)
    reference := collection.Reference(document)
}

Documents can also be removed:

func main() {
    store, err := leia.NewStore("my.db")
    collection := store.Collection("credentials")
    ...
    
    // define your document
    document := leia.DocumentFromString("{...some json...}")
    
    // remove a document using a leia.Document
    err := collection.Delete(document)
}
Reading

A document can be retrieved by reference:

func main() {
    store, err := leia.NewStore("my.db")
    collection := store.Collection("credentials")
    ...
    
    // document by reference, it returns nil when not found
    document, err := collection.Get(reference)
}
Searching

The major benefit of leia is searching. The performance of a search greatly depends on the available indices on a collection. If no index matches the query, a bbolt cursor is used to loop over all documents in the collection.

Leia supports equal, prefix and range queries. The first argument for each matcher is the JSON path using the syntax from gjson. Only basic path syntax is used. There is no support for wildcards or comparison operators. The second argument is the value to match against. Leia can only combine query terms using AND logic.

func main() {
    ...
    
    // define a new query
    query := leia.New(leia.Eq("subject", "some_value")).
                  And(leia.Range("some.path.#.amount", 1, 100))
}

Getting results can be done with either Find or Iterate. Find will return a slice of documents. Iterate will allow you to pass a DocWalker which is called for each hit.

func main() {
    ...
    
    // get a slice of documents
    documents, err := collection.Find(query)
    
    // use a DocWalker
    walker := func(ref []byte, doc []byte) error {
    	// do something with the document
    }
    err := collection.Iterate(query, walker)
}

Indexing

Indexing JSON documents is where the real added value of leia lies. For each collection multiple indices can be added. Each added index will slow down write operations.

An index can be added and removed:

func main() {
    ...
    
    // define the index
    index := leia.NewIndex("compound",
                leia.NewFieldIndexer("subject"),
                leia.NewFieldIndexer("some.path.#.amount"),
    )
    
    // add it to the collection
    err := collection.AddIndex(index)
    
    // remove it from the collection
    err := collection.DropIndex("compound")
}

The argument for NewFieldIndexer uses the same notation as the query parameter, also without wildcards or comparison operators. Adding an index will trigger a re-index of all documents in the collection. Adding an index with a duplicate name will ignore the index.

Alias option

Leia support indexing JSON paths under an alias. An alias can be used to index different documents but use a single query to find both.

func main() {
    ...
    
    // define the index for credentialX
    indexX := leia.NewIndex("credentialX", leia.NewFieldIndexer("credentialSubject.id", leia.AliasOption{Alias: "subject"}))
    // define the index for credentialY
    indexY := leia.NewIndex("credentialY", leia.NewFieldIndexer("credentialSubject.organization.id", leia.AliasOption{Alias: "subject"}))
    
    ...

    // define a new query
    query := leia.New(leia.Eq("subject", "some_value"))
}

The example above defines two indices to a collection, each index has a different JSON path to be indexed. Both indices will be used when the given query is executed, resulting in documents that match either index.

Transform option

A transformer can be defined for a FieldIndexer. A transformer will transform the indexed value and query parameter. This can be used to allow case-insensitive search or add a soundex style index.

func main() {
    ...
    
    // This index transforms all values to lowercase
    index := leia.NewIndex("credential", leia.NewFieldIndexer("subject", leia.TransformOption{Transform: leia.ToLower}))
    
    ...

    // these queries will yield the same result
    query1 := leia.New(leia.Eq("subject", "VALUE"))
    query2 := leia.New(leia.Eq("subject", "value"))
}
Tokenizer option

Sometimes JSON fields contain a whole text. Leia has a tokenizer option to split a value at a JSON path into multiple keys to be indexed. For example, the sentence "The quick brown fox jumps over the lazy dog" could be tokenized so the document can easily be found when the term fox is used in a query. A more advanced tokenizer could also remove common words like the.

func main() {
    ...
    
    // This index transforms all values to lowercase
    index := leia.NewIndex("credential", leia.NewFieldIndexer("text", leia.TokenizerOption{Tokenizer: leia.WhiteSpaceTokenizer}))
    
    ...

    // will match {"text": "The quick brown fox jumps over the lazy dog"}
    query := leia.New(leia.Eq("subject", "fox"))
}

All options can be combined.

Documentation

Index

Constants

View Source 
const KeyDelimiter = 0x10

Variables

View Source 
var ErrInvalidJSON = errors.New("invalid json")

ErrInvalidJSON is returned when invalid JSON is parsed

View Source 
var ErrNoIndex = errors.New("no index found")

ErrNoIndex is returned when no index is found to query against

View Source 
var ErrNoQuery = errors.New("no query given")

ErrNoQuery is returned when an empty query is given

Functions

func ToLower 

func ToLower(terms interface{}) interface{}

ToLower transforms all Unicode letters mapped to their lower case. It only transforms objects that conform to the Stringer interface.

func WhiteSpaceTokenizer 

func WhiteSpaceTokenizer(text string) []string

WhiteSpaceTokenizer tokenizes the string based on the /\S/g regex

Types

type Collection 

type Collection interface {
	// AddIndex to this collection. It doesn't matter if the index already exists.
	// If you want to override an index (by name) drop it first.
	AddIndex(index ...Index) error
	// DropIndex by name
	DropIndex(name string) error
	// Add a set of documents to this collection
	Add(jsonSet []Document) error
	// Get returns the data for the given key or nil if not found
	Get(ref Reference) (*Document, error)
	// Delete a document
	Delete(doc Document) error
	// Find queries the collection for documents
	// returns ErrNoIndex when no suitable index can be found
	// returns context errors when the context has been cancelled or deadline has exceeded.
	// passing ctx prevents adding too many records to the result set.
	Find(ctx context.Context, query Query) ([]Document, error)
	// Reference uses the configured reference function to generate a reference of the function
	Reference(doc Document) Reference
	// Iterate over documents that match the given query
	Iterate(query Query, walker DocumentWalker) error
	// IndexIterate is used for iterating over indexed values. The query keys must match exactly with all the FieldIndexer.Name() of an index
	// returns ErrNoIndex when no suitable index can be found
	IndexIterate(query Query, fn ReferenceScanFn) error
}

Collection defines a logical collection of documents and indices within a store.

type Document 

type Document struct {
	// contains filtered or unexported fields
}

Document represents a JSON document in []byte format

func DocumentFromBytes 

func DocumentFromBytes(json []byte) Document

DocumentFromBytes creates a Document from a JSON string

func DocumentFromString 

func DocumentFromString(json string) Document

DocumentFromString creates a Document from a JSON string

func (Document) Bytes 

func (d Document) Bytes() []byte

Bytes returns the raw bytes

func (Document) KeysAtPath 

func (d Document) KeysAtPath(jsonPath string) ([]Key, error)

KeysAtPath returns the values found at the JSON path query as Key

func (Document) ValuesAtPath 

func (d Document) ValuesAtPath(jsonPath string) ([]interface{}, error)

ValuesAtPath returns a slice with the values found at the given JSON path query

type DocumentWalker 

type DocumentWalker func(key Reference, value []byte) error

DocumentWalker defines a function that is used as a callback for matching documents. The key will be the document Reference (hash) and the value will be the raw document bytes

type FieldIndexer 

type FieldIndexer interface {
	// Name is used for matching against a Query
	Name() string
	// Keys returns the keys that matched this document. Multiple keys are combined by the index
	Keys(document Document) ([]Key, error)
	// Tokenize may split up Keys and search terms. For example split a sentence into words.
	Tokenize(value interface{}) []interface{}
	// Transform is a function that alters the value to be indexed as well as any search criteria.
	// For example LowerCase is a Transform function that transforms the value to lower case.
	Transform(value interface{}) interface{}
}

FieldIndexer is the public interface that defines functions for a field index instruction. A FieldIndexer is used when a document is indexed.

func NewFieldIndexer 

func NewFieldIndexer(jsonPath string, options ...IndexOption) FieldIndexer

NewFieldIndexer creates a new fieldIndexer leave the name empty to use the json path as name. the name is to be used as query key when searching

type Index 

type Index interface {
	// Name returns the name of this index
	Name() string

	// Add indexes the document. It uses a sub-bucket of the given bucket.
	// It will only be indexed if the complete index matches.
	Add(bucket *bbolt.Bucket, ref Reference, doc Document) error

	// Delete document from the index
	Delete(bucket *bbolt.Bucket, ref Reference, doc Document) error

	// IsMatch determines if this index can be used for the given query. The higher the return value, the more likely it is useful.
	// return values lie between 0.0 and 1.0, where 1.0 is the most useful.
	IsMatch(query Query) float64

	// Iterate over the key/value pairs given a query. Entries that match the query are passed to the iteratorFn.
	// it will not filter out double values
	Iterate(bucket *bbolt.Bucket, query Query, fn iteratorFn) error

	// BucketName returns the bucket name for this index
	BucketName() []byte

	// Sort the query so its parts align with the index parts.
	// includeMissing, if true, the sort will append queryParts not matched by an index at the end.
	Sort(query Query, includeMissing bool) []QueryPart

	// QueryPartsOutsideIndex selects the queryParts that are not covered by the index.
	QueryPartsOutsideIndex(query Query) []QueryPart

	// Depth returns the number of indexed fields
	Depth() int
}

Index describes an index. An index is based on a json path and has a name. The name is used for storage but also as identifier in search options.

func NewIndex 

func NewIndex(name string, parts ...FieldIndexer) Index

NewIndex creates a new blank index. If multiple parts are given, a compound index is created.

type IndexOption 

type IndexOption func(fieldIndexer *fieldIndexer)

IndexOption is the option function for adding options to a FieldIndexer

func AliasOption 

func AliasOption(alias string) IndexOption

AliasOption is the option for a FieldIndexer to add a custom JSON path that will also resolve to the same Index part

func TokenizerOption 

func TokenizerOption(tokenizer Tokenizer) IndexOption

TokenizerOption is the option for a FieldIndexer to split a value to be indexed into multiple parts. Each part is then indexed separately.

func TransformerOption 

func TransformerOption(transformer Transform) IndexOption

TransformerOption is the option for a FieldIndexer to apply transformation before indexing the value. The transformation is also applied to a query value that matches the indexed field.

type Key 

type Key []byte

Key is used as DB key type

func ComposeKey 

func ComposeKey(current Key, additional Key) Key

ComposeKey creates a new key from two keys

func KeyOf 

func KeyOf(value interface{}) Key

KeyOf creates a key from an interface

func (Key) Split 

func (k Key) Split() []Key

Split splits a compound key into parts

func (Key) String 

func (k Key) String() string

String returns the string representation, only useful if a Key represents readable bytes

type Query 

type Query interface {
	// And adds a condition to query on
	And(part QueryPart) Query

	// Parts returns the different parts of the query
	Parts() []QueryPart
}

func New 

func New(part QueryPart) Query

New creates a new query with an initial query part. Both begin and end are inclusive for the conditional check.

type QueryPart 

type QueryPart interface {

	// Name returns the name that matches fieldIndexer.Name() so actually the alias or JSON path
	Name() string

	// Seek returns the key for cursor.Seek
	Seek() (Key, error)

	// Condition returns true if given key falls within this condition.
	// The optional transform fn is applied to this query part before evaluation is done.
	Condition(key Key, transform Transform) (bool, error)
}

func Eq 

func Eq(name string, value interface{}) QueryPart

Eq creates a query part for an exact match

func Prefix 

func Prefix(name string, value interface{}) QueryPart

Prefix creates a query part for a partial match The beginning of a value is matched against the query.

func Range 

func Range(name string, begin interface{}, end interface{}) QueryPart

Range creates a query part for a range query

type Reference 

type Reference []byte

Reference equals a document hash. In an index, the values are references to docs.

func (Reference) ByteSize 

func (r Reference) ByteSize() int

ByteSize returns the size of the reference, eg: 32 bytes for a sha256

func (Reference) EncodeToString 

func (r Reference) EncodeToString() string

EncodeToString encodes the reference as hex encoded string

type ReferenceFunc 

type ReferenceFunc func(doc Document) Reference

ReferenceFunc is the func type used for creating references. references are the key under which a document is stored. a ReferenceFunc could be the sha256 func or something that stores document in chronological order. The first would be best for random access, the latter for chronological access

type ReferenceScanFn 

type ReferenceScanFn func(key []byte, value []byte) error

ReferenceScanFn is a function type which is called with an index key and a document Reference as value

type Store 

type Store interface {
	// Collection creates or returns a collection.
	// On the db level it's a bucket for the documents and 1 bucket per index.
	Collection(name string) Collection
	// Close closes the bbolt DB
	Close() error
}

Store is the main interface for storing/finding documents

func NewStore 

func NewStore(dbFile string, noSync bool) (Store, error)

NewStore creates a new store. the noSync option disables flushing to disk, ideal for testing and bulk loading

type Tokenizer 

type Tokenizer func(string) []string

Tokenizer is a function definition that transforms a text into tokens

type Transform 

type Transform func(interface{}) interface{}

Transform is a function definition for transforming values and search terms.

Source Files

View all Source files

Directories

Path Synopsis
examples
options
vcs

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.