heptane

README

HEPTANE - cacHEd disPerse Table bAckeNd framEwork

Heptane is a framework for development of backend solutions that store data in disperse databases:

• It provides a uniform API to access the data on disperse tables.
• It provides several implementations of the given API using different providers for the tables and the cache, i.e. cassandra, hbase, dynamodb, sharding over a cluster of relational databases, memcached, redis, redis cluster.
• It provides development tools.

The API is defined with the following goals:

• Only the common operations to all disperse database providers are implemented.
• Different database and cache providers can be used at the same time.
• High enphasis on the parallel access to the database and cache.

In detail:

• Users define disperse table metadata: name, fields, primary key, partition key, cache policies.
• The framework provides an API to write data into the tables specified by the metadata, updating the cache at the same time.
• The framework provides an API to read data from the tables specified by the metadata, using the cache.
• The framework takes care of the particular implementation details of each db provider.

The framework provides the following development tools:

• Given table metadata, the framework is able to create the tables in the database provider.
• Given an existing schema in the database provide, the framework is able to extract the table metadata.
• The framework is able to migrate data between different database providers.

Installation

Using go get as usual:

• go get github.com/heptanes/heptane
• import "github.com/heptanes/heptane"

Documentation

Overview

HEPTANE - cacHEd disPerse Table bAckeNd framEwork.

This package provides a main interface Heptane and one implementation accessible from the constructor New().

Heptane maintains metadata about a set of tables (an instance of Table, an instance of RowProvider and an optional instance of CacheProvider per table) and provides operations to access the contents of these tables.

The Table

The type Table describes the underlying physical table:

the Name,

the names of the fields that compose the PartitionKey (there must be at least one field),

the names of the fields that compose the PrimaryKey (the PrimaryKey must contain the PartitionKey as prefix),

the names of the fields that compose the Values (there may be no fields at all),

the types of each field in a map FieldTypesByName (strings and bools),

the prefix PrimaryKeyCachePrefix in an external cache that contains the PrimaryKeys as cache key and the Values as cache values.

RowProviders And CacheProviders

Operations are performed on TableNames, then Heptane looks for the registered RowProvider and CacheProvider for that TableName, performs the action on the underlying table through the RowProvider and synchronizes the PrimaryKey cache through the CacheProvider.

RowProviders offer access to specific databases. The definition of a Table is that of a disperse table, databases that implement disperse tables are a natural fit for a RowProvider implementation. The API is simple enough that RowProvider implementations for relational databases are possible too.

CacheProviders are simple key value pairs that need to provide only the operations Set and Get. Values are stored without expiration time.

Accesses

The type Access is the interface for all operations, and there is one struct for each operation: Create, Retrieve, Update and Delete.

Create sends a RowCreate operation to the RowProvider and, if successful, sends a CacheSet operation to the CacheProvider.

When the full PrimaryKey is given, Retrieve sends a CacheGet operation to the CacheProvider. The row is returned if found, otherwise it sends a RowRetrieve operation to the RowProvider, and then sends a CacheSet operation to the CacheProvider.

When the full PrimaryKey is not given, but at least the full PartitionKey must be given, Retrieve sends a RowRetrieve operation to the RowProvider, and then sends a CacheSet operation to the CacheProvider for each retrieved row.

When the full Values are given, Update sends a RowUpdate operation to the RowProvider and, if successful, sends a CacheSet operation to the CacheProvider.

When the full Values are not given, Update sends the partial RowUpdate operation to the RowProvider and, if successful, sends a RowRetrieve to the RowProvider and, if successful, it sends a CacheSet operation to the CacheProvider.

Delete sends a RowDelete operation to the RowProvider and, if successful, sends a CacheSet operation to the CacheProvider. The value stored in the cache means that there is no row in the table, so the next Retrieve does not need to query the RowProvider.

Retrieve must be passed as reference so the RetrievedValues set by Heptane may be read by the client code.

Index

• type Access
• type CacheProviderAccessError
  • func (e CacheProviderAccessError) Error() string
• type Create
• type Delete
• type Heptane
  • func New() Heptane
• type MissingFieldValueError
  • func (e MissingFieldValueError) Error() string
• type MultipleErrors
  • func (e MultipleErrors) Error() string
• type NullRowProviderError
  • func (e NullRowProviderError) Error() string
• type Retrieve
• type RowProviderAccessError
  • func (e RowProviderAccessError) Error() string
• type UnregisteredTableError
  • func (e UnregisteredTableError) Error() string
• type UnsupportedAccessTypeError
  • func (e UnsupportedAccessTypeError) Error() string
• type UnsupportedFieldValueError
  • func (e UnsupportedFieldValueError) Error() string
• type Update

Types

type Access

type Access interface{}

Access is the interface of all types that represent an access to a table using a cache.

type CacheProviderAccessError

type CacheProviderAccessError struct {
	Access c.CacheAccess
	Err    error
}

CacheProviderAccessError is produced when a CacheProvider returns an error for a given CacheAccess.

func (CacheProviderAccessError) Error

func (e CacheProviderAccessError) Error() string

type Create

type Create struct {
	// TableName is the name of the table.
	TableName r.TableName
	// FieldValues contains the primary key and the values.
	FieldValues r.FieldValuesByName
}

Create specifies the creation of a row in a table.

type Delete

type Delete struct {
	// TableName is the name of the table.
	TableName r.TableName
	// FieldValues contains the primary key.
	FieldValues r.FieldValuesByName
}

Delete specifies the deletion of a row in a table.

type Heptane

type Heptane interface {
	// Register creates and updates a mapping between a TableName and its
	// specification: the Table, RowProvider and CacheProvider.
	Register(r.Table, r.RowProvider, c.CacheProvider) error
	// Unregister deletes the mapping between the given Tablename and its
	// specification.
	Unregister(r.TableName)
	// TableNames returns a slice of all names of tables that have been
	// registered.
	TableNames() []r.TableName
	// Table returns the current associated Table of the given TableName.
	Table(r.TableName) r.Table
	// RowProvider the current associated RowProvider of the given
	// TableName.
	RowProvider(r.TableName) r.RowProvider
	// CacheProvider the current associated CacheProvider of the given
	// TableName.
	CacheProvider(r.TableName) c.CacheProvider
	// Access performs the given acccess to the table using the cache.
	Access(Access) error
	// AccessSlice performs several acccesses to the table using the cache.
	AccessSlice([]Access) []error
}

Heptane is the main interface, it provides a uniform access to tables supported by different RowProviders and CacheProviders. Safe to be used from different goroutines.

func New

func New() Heptane

New returns a new instance of Heptane.

type MissingFieldValueError

type MissingFieldValueError struct {
	TableName         r.TableName
	FieldName         r.FieldName
	FieldValuesByName r.FieldValuesByName
}

MissingFieldValueError is produced when a Table defines a FieldName that has no value in a FieldValuesByName.

func (MissingFieldValueError) Error

func (e MissingFieldValueError) Error() string

type MultipleErrors

type MultipleErrors struct {
	Errors []error
}

MultipleErrors encapsulates one or more errors typically produced concurrently.

func (MultipleErrors) Error

func (e MultipleErrors) Error() string

type NullRowProviderError

type NullRowProviderError struct {
	TableName r.TableName
}

NullRowProviderError is produced when a Table is registered with a nil RowProvider.

func (NullRowProviderError) Error

func (e NullRowProviderError) Error() string

type Retrieve

type Retrieve struct {
	// TableName is the name of the table.
	TableName r.TableName
	// FieldValues contains the partition key and optionally other fields
	// from the primary key.
	FieldValues r.FieldValuesByName
	// FieldValues will contain one or more rows, each one with all its
	// fields.
	RetrievedValues []r.FieldValuesByName
}

Retrieve specifies the retrieval of one or several rows in a table.

type RowProviderAccessError

type RowProviderAccessError struct {
	Access r.RowAccess
	Err    error
}

RowProviderAccessError is produced when a RowProvider returns an error for a given RowAccess.

func (RowProviderAccessError) Error

func (e RowProviderAccessError) Error() string

type UnregisteredTableError

type UnregisteredTableError struct {
	TableName r.TableName
}

UnregisteredTableError is produced when an Access is tried on a TableName that has not been registered.

func (UnregisteredTableError) Error

func (e UnregisteredTableError) Error() string

type UnsupportedAccessTypeError

type UnsupportedAccessTypeError struct {
	Access Access
}

UnsupportedAccessTypeError is produced when the type of an Access is not supported. Current supported types are Create, Retrieve, Update and Delete.

func (UnsupportedAccessTypeError) Error

func (e UnsupportedAccessTypeError) Error() string

type UnsupportedFieldValueError

type UnsupportedFieldValueError struct {
	FieldType  r.FieldType
	FieldValue r.FieldValue
}

UnsupportedFieldValueError is produced when the type of a FieldValue does not match the corresponding FieldType.

func (UnsupportedFieldValueError) Error

func (e UnsupportedFieldValueError) Error() string

type Update

type Update struct {
	// TableName is the name of the table.
	TableName r.TableName
	// FieldValues contains the primary key and the values.
	FieldValues r.FieldValuesByName
}

Update specifies the update of a row in a table.