sdsshared

README

Simple Data Service - shared library

This package presents code to be used by each data resource in order to have a standard API used by every service.

Code Organisation

The following approach is used:

1. Application domain types go in the root - api.go, server.go, util.go, etc
2. Implementation of the application domain go in subpackages - badger, sqlite, etc
3. Everything is tied together in the cmd subpackages - cmd/shared

There is no internal package here as there need be no hiding of code from implementation code.

Usage

Add to project

go get github.com/rhythmicsound/sdsshared

Then you can create a new service by picking your connector and running something like:

package main

import (
	"fmt"
	"log"

	sdsshared "github.com/RhythmicSound/sds-shared"
	badgerconnector "github.com/RhythmicSound/sds-shared/badgerConnector"
)

func main() {

	connector := badgerconnector.New(sdsshared.ResourceServiceName, sdsshared.DatasetURI)

	log.Fatalln(sdsshared.StartServer(connector, "", 0))
}

Using default type values for the arguments to StartServer allows service name and ports to be set using environment variables at runtime.

An example execute command is:

debug=false \
name="postcodeUK-Service" \
database_uri="working/databases/postcodeUKdb" \
dataset_uri="https://storage.cloud.google.com/simple-data-service/datasets/postcodesUK.zip" \ 
objectname="datasets/postcodesUK.zip" \
go run cmd/dummy.go

Settings

Settings for services created from this library can be hardcoded or set using environment variables

Variable	Explanation	Default
debug	Whether to print verbose output to log and load test data to database. Not for use in production	"false"
database_uri	The path -URL or local path- to the database resource to connect to.	"working/databases/simpledataservice-default/" (N.B. this points at a directory as BadgerDB is the default db in use. This could be a URL or path to local file. In some instances, if no db exists in the path given, one could be created.)
dataset_uri	The path -URL or local path- to the dataset resource used to rebuild the database.	"working/datasets/data.zip"
bucket	The cloud bucket from which to find the dataset archive. (Required only if downloading the dataset from behind an authentication wall)	"simple-data-service"
object	The cloud object name found in DatasetBucketName that identifies the dataset archive for download. (Required only if downloading the dataset from behind an authentication wall)	-
name	The name of this service as visible to other services.	"Default Resource Name"
publicport	PublicPort is the port from which this API can be accessed for data retrieval	"8080"
downloaddir	The local path where download files will be saved to	"working/downloads"

Writing new backend storage connectors

Implement DataResource interface

Example:

type impl struct{
	ResourceName string
	Database     *badger.DB
	versioner    sdsshared.VersionManager
}

func NewImpl(resourceName, datasetArchiveLocation string) *impl{
  return &impl{
    	ResourceName: resourceName,
		versioner: sdsshared.VersionManager{
			Repo:           sdsshared.DatasetURI,
			LastUpdated:    "",
			CurrentVersion: 0,
			DataSources:    make([]string, 0),
		},
  }
}

(im impl) Startup()error{
  ... //Startup scripts
}

vt := &VersionManager{
    Repo = repoURIAddress
}

(im impl) UpdateDataset(vt)(*VersionManager,error){
  ... // Logic to keep the database synced with a master versioned dataset archive somehwere

    //Passing a VersionManager as an arg should overwrite internal VM created in New. 
    //Must accept nil to use default
}

(im impl) Retrieve(someSearchString)(sdsshared.SimpleData,error){
   ... //Fetching data from the database based on given key value. 
    //Use the latest VersionManager to complete the Meta elements of SimpleData response object
}

(im impl) Shutdown()error{
  ... //Shutdown scripts
}

See the badgerConnector package for best practise

Documentation

Index

• Variables
• func CreateKVStoreKey(key string, sep string) string
• func GCPDownload(bucket, object string) error
• func GetEnv(variable, deflt string) string
• func NewHTTPClient() *http.Client
• func StartServer(dr DataResource, serverName string, port int) error
• type DataOutput
• type DataResource
• type DataResourceImplementorTemplate
• type Meta
• type SimpleData
• type VersionManager
  • func (vt *VersionManager) UpdateDataset(dr DataResource) error

Variables

var (
	//ResourceServiceName is the name of this service as visible to other services.
	//
	//Passing value through to StartServers function overrides this by hard coding.
	// To avoid this pass an empty string into StartServers
	// function or repass this value as sdsshared.ResourceServiceName.
	ResourceServiceName string
	//Whether verbose logging and test data should be used.
	// Never set true for production
	DebugMode bool
	//DBURI is the path -URL or local path- to the database resource.
	// If it doesn't exist one will be created at this resource for
	// some connector implementations. For local dbs this may be pre/suf-fixed
	// to allow for dataset updates with minimised downtime
	DBURI string
	//DBURI is the path (URL or local path) to the archive file for
	//the dataset used to rebuild the database
	DatasetURI string
	//PublicPort is the port from which this API can be accessed for data retrieval
	PublicPort string
	//LocalDownloadDir is the local relative or absolute path to a downloads folder to use
	LocalDownloadDir string

	//DatasetBucketName is the cloud bucket from which to find the dataset archive
	DatasetBucketName = "simple-data-service"
	//DatasetObjectName is the cloud object name found in DatasetBucketName that
	// identifies the dataset archive for download
	DatasetObjectName string
)

Functions

func CreateKVStoreKey

func CreateKVStoreKey(key string, sep string) string

CreateKVStoreKey creates a standard key for use in kv storage dbs. Appends a Unix timestamp so that duplicate entries can be stored seperately, sorted easily, and deduped using proper protocols without requiring calls to meta

Using a Unix timestamp means just sorted the keys in place will give time sorted list

func GCPDownload

func GCPDownload(bucket, object string) error

GCPDownload downloads assets from Google Cloud Storage with the given

Object name and within the given Bucket

func GetEnv

func GetEnv(variable, deflt string) string

GetEnvar get an environment variable and if it is black sets the given default.

The value may be set as default string purposefully rather than omitted. In this case an empty string will be returned instead of the default string.

func NewHTTPClient

func NewHTTPClient() *http.Client

NewHTTPClient is the default client to be used instead of default client. Ammended timeouts. See //https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

func StartServer

func StartServer(dr DataResource, serverName string, port int) error

StartServer runs the server to interface with the system using the api methods of DataResource.

If port is set to 0 a default setting or the setting given using environment variable `publicport` will be used

Similarly if serverName is not set the default will be used or the value in environment variable `name` suffixed with the word 'server'

Types

type DataOutput

type DataOutput struct {
	//... Requested data schema...
	Values interface{} `json:"values"`
}

DataOutputis the SimpleData componant with the requested data payload

type DataResource

type DataResource interface {
	Startup() error
	UpdateDataset() (VersionManager, error)
	//Retrieve takes query token and map[string]string group of query args
	// received in the GET request.
	// Returned []byte is JSON representation of SimpleData
	Retrieve(string, map[string]string) (SimpleData, error)
	Shutdown() error
}

DataResource is the interface each Resource service uses and is a central library unit used for a centralised server facility that handles JWT checking centrally.

type DataResourceImplementorTemplate

type DataResourceImplementorTemplate struct {
	DatabaseHandle interface{}
	// contains filtered or unexported fields
}

DataResourceImplementorTemplate is a simple outline of the basic structure that can implement the full DataResource interface. See `badgerdb` for best practise

type Meta

type Meta struct {
	Resource    string   `json:"resource"`
	LastUpdated string   `json:"dataset_updated,omitempty"` //time.RFC3339
	DataSources []string `json:"data_sources,omitempty"`
}

Meta is the SimpleData componant with meta data on the datasource being queried

type SimpleData

type SimpleData struct {
	ResultCount    int               `json:"result_count"`
	RequestOptions map[string]string `json:"request_options,omitempty"`
	Meta           Meta              `json:"meta"`
	Data           DataOutput        `json:"data"`
	Errors         map[string]string `json:"errors,omitempty"`
}

SimpleData is the standard response interface to the end user. Data resources should return

type VersionManager

type VersionManager struct {
	CurrentVersion string `json:"version,omitempty"`
	//Repo is where to get the dataset archive from (URL)
	Repo string `json:"repo,omitempty"`
	//LastUpdated is when the Repo was last updated from latest version (time.RFC3339)
	LastUpdated string `json:"dataset_updated"`
	//List of initial data sources gained from last update from repo
	DataSources []string `json:"data_sources"`
}

VersionManager is the struct that allows the instance to check its current used dataset version and compare it against the latest available to judge update need

func (*VersionManager) UpdateDataset

func (vt *VersionManager) UpdateDataset(dr DataResource) error