storage

README

Storage  

Unified key/value storage interface for several backing technologies.

• Bolt (BoltDB backed store; production ready)
• Memcached (Memcached backed store; production ready)
• Redis (Redis backed store; production ready)
• S3 (AWS S3 backed store; production ready)
• DynamoDB (AWS DynamoDB backed store; production ready)
• AWS ElasticSearch([AWS ElasticSearch] (https://aws.amazon.com/elasticsearch-service/) backed store; production ready)
• Folder (local folder backed store; intended for dev)
• Local (application's memory backed store; intended for dev)

Usage and Examples

go get -u gopkg.in/kyani-inc/storage.v1

You can write your application to use the storage interface and then change out the backing technology based on environment and application needs. For example, running Redis in production and using Local or Folder for local development.

package main

import (
	"fmt"
	"os"
	"strings"

	"gopkg.in/kyani-inc/storage.v1"
)

var store storage.Storage 

func init() {
	var err error

	switch os.Getenv("STORAGE_ENGINE") {
	case "s3":
		secret := os.Getenv("AWS_SECRET_KEY")
		access := os.Getenv("AWS_ACCESS_KEY")
		bucket := os.Getenv("AWS_BUCKET")
		region := os.Getenv("AWS_REGION")
		content := "application/json; charset=utf-8"
		prefix := "test/storage"

		store, err = storage.S3(access, secret, bucket, region, content, prefix)
		// all keys will be surrounded by the prefix value and content
		// extension (if recognized) like: "test/storage/name.json"

	case "folder":
		store, err = storage.Folder(os.Getenv("FOLDER_STORAGE_PATH"))

	case "redis":
		store = storage.Redis(os.Getenv("REDIS_HOST"), os.Getenv("REDIS_PORT"))

	case "memcached":
		hosts := strings.Split(os.Getenv("MEMCACHED_HOSTS"), ",")
		store = storage.Memcached(hosts)

	case "bolt":
		store, err = storage.Bolt(os.Getenv("BOLTDB_FILE_PATH"))

	case "dynamodb":
		secret := os.Getenv("AWS_SECRET_KEY")
		access := os.Getenv("AWS_ACCESS_KEY")
		region := os.Getenv("AWS_REGION")
		table := os.Getenv("DYNAMODB_TABLE")

		store, err = storage.DynamoDB(access, secret, region, table)
        
    case "elasticsearch":
		host := os.Getenv("ES_HOST")
		scheme := os.Getenv("ES_SCHEME")
		index := os.Getenv("ES_INDEX")
		namespace := os.Getenv("APP_NAME")
		awsSecret := os.Getenv("AWS_SECRET_KEY")
		awsKey := os.Getenv("AWS_ACCESS_KEY")

		store, err = storage.ElasticSearch(host, scheme, index, namespace, awsKey, awsSecret)

	default:
		store = storage.Local()
	}

	if err != nil {
		// Handle the error appropriately.
		// You may not want to panic in your application.
		panic(err.Error())
	}
}

func main() {
	// This is designed for XML or JSON documents but any []byte can be used.
	err := store.Put("name", []byte("John Doe"))

	if err != nil {
		fmt.Println("Error saving name", err.Error())
		return
	}

	data := store.Get("name") // []byte("John Doe")

	if data == nil {
		fmt.Println("Missing key: name");
		return
	}

	fmt.Printf("Hello, %s.\n", data) // Hello, John Doe.

	store.Delete("name") // remove "name"

	store.Flush() // get rid of everything
}

Testing

run go test -v ./...

Env Vars for Testing

To test certain drivers you will need to provide the services to test against. The tests are built using gotenv so that you can provide environment variables for the tests. Otherwise tests that require these variables will be skipped.

For example: Testing the Redis implementation on your local machine would require the file providers/redis/.env with the following contents (replace HOST and PORT with your values).

REDIS_HOST=127.0.0.1
REDIS_PORT=6379

Documentation

Overview

Package storage creates an interface for several key/value storage technologies.

Index

• type Storage
  • func Bolt(path string) (Storage, error)
  • func DynamoDB(access string, secret string, region string, table string) (Storage, error)
  • func ElasticSearch(host, scheme, index, namespace, awsKey, awsSecret string) (Storage, error)
  • func Folder(path string) (Storage, error)
  • func Local() Storage
  • func Memcached(hosts []string) Storage
  • func Redis(host, port string) Storage
  • func S3(access, secret, bucket, region, content, prefix string) (Storage, error)

Types

type Storage

type Storage interface {
	// Get returns data by key. Missing keys return `nil`.
	Get(key string) []byte

	// Put will overwrite data by key.
	Put(key string, data []byte) error

	// Delete will attempt to remove a value by key. Idempotent.
	Delete(key string)

	// Flush clears all keys from the storage. Idempotent.
	Flush()
}

Storage represents a storage facility agnostic of the backing technology.

func Bolt

func Bolt(path string) (Storage, error)

Bolt utilizes a BoltDB database (https://github.com/boltdb/bolt) for storage.

func DynamoDB

func DynamoDB(access string, secret string, region string, table string) (Storage, error)

AWS DynamoDB storage.

func ElasticSearch

func ElasticSearch(host, scheme, index, namespace, awsKey, awsSecret string) (Storage, error)

ElasticSearch storage

func Folder

func Folder(path string) (Storage, error)

Folder uses the application's underlying file structure for storage.

func Local

func Local() Storage

Local uses the applications memory for storage.

func Memcached

func Memcached(hosts []string) Storage

Memcache uses one or more Memcache hosts for storage.

func Redis

func Redis(host, port string) Storage

Redis uses a Redis instance for storage.

func S3

func S3(access, secret, bucket, region, content, prefix string) (Storage, error)

S3 uses Amazon AWS S3 for storage.

Every key combined with the prefix (to support Flush) and a possible extension determined by content. The field content is the content type to use with all keys. For example: "application/json; charset=utf-8".