remember

README

⭐   the project to show your appreciation. :arrow_upper_right:

Cache Slow Database Queries

This package is used to cache the results of slow database queries in memory or Redis. It can be used to cache any form of data (eg. function memoization). A Redis and in-memory storage driver is provided.

See Article for further details including a tutorial.

The package is production ready and the API is stable. A variant of this package has been used in production for over 4 years.

Installation

go get -u github.com/rocketlaunchr/remember-go

Create a Key

Let’s assume the query’s argument is an arbitrary search term and a page number for pagination.

CreateKeyStruct

CreateKeyStruct can generate a JSON based key by providing a struct.

type Key struct {
    Search string
    Page   int `json:"page"`
}

var key string = remember.CreateKeyStruct(Key{"golang", 2})

CreateKey

CreateKey provides more flexibility to generate keys:

// Key will be "search-golang-2"
key :=  remember.CreateKey(false, "-", "search-x-y", "search", "golang", 2)

Initialize the Storage Driver

In-Memory

import "github.com/rocketlaunchr/remember-go/memory"

var ms = memory.NewMemoryStore(10 * time.Minute)

Redis

The Redis storage driver relies on Gary Burd’s excellent Redis client library.

import red "github.com/rocketlaunchr/remember-go/redis"
import "github.com/gomodule/redigo/redis"

var rs = red.NewRedisStore(&redis.Pool{
    Dial: func() (redis.Conn, error) {
        return redis.Dial("tcp", "localhost:6379")
    },
})

Memcached

An experimental (and untested) memcached driver is provided. It relies on Brad Fitzpatrick's memcache driver.

Ristretto

DGraph's Ristretto is a fast, fixed size, in-memory cache with a dual focus on throughput and hit ratio performance.

Nocache

This driver is for testing purposes. It does not cache any data.

Create a SlowRetrieve Function

The package initially checks if data exists in the cache. If it doesn’t, then it elegantly fetches the data directly from the database by calling the SlowRetrieve function. It then saves the data into the cache so that next time it doesn’t have to refetch it from the database.

type Result struct {
    Title string
}

slowQuery := func(ctx context.Context) (interface{}, error) {
    results := []Result{}

    stmt := `
        SELECT title
        FROM books WHERE title LIKE ?
        ORDER BY title LIMIT ?, 20
    `

    rows, _ := db.QueryContext(ctx, stmt, search, (page-1)*20)

    for rows.Next() {
        var title string
        rows.Scan(&title)
        results = append(results, Result{title})
    }

    return results, nil
}

Usage

key := remember.CreateKeyStruct(Key{"golang", 2})
exp := 10*time.Minute

results, found, err := remember.Cache(ctx, ms, key, exp, slowQuery, remember.Options{GobRegister: false})

return results.([]Result) // Type assert in order to use

Gob Register Errors

The Redis storage driver stores the data in a gob encoded form. You have to register with the gob package the data type returned by the SlowRetrieve function. It can be done inside a func init(). Alternatively, you can set the GobRegister option to true. This will impact concurrency performance and is thus not recommended.

Other useful packages

• awesome-svelte - Resources for killing react
• dataframe-go - For statistics, machine-learning, and data manipulation/exploration
• dbq - Zero boilerplate database operations for Go
• electron-alert - SweetAlert2 for Electron Applications
• google-search - Scrape google search results
• igo - A Go transpiler with cool new syntax such as fordefer (defer for for-loops)
• mysql-go - Properly cancel slow MySQL queries
• react - Build front end applications using Go
• testing-go - Testing framework for unit testing

Legal Information

The license is a modified MIT license. Refer to LICENSE file for more details.

© 2019-21 PJ Engineering and Business Solutions Pty. Ltd.

Final Notes

Feel free to enhance features by issuing pull-requests.

Documentation

Index

• func Cache(ctx context.Context, c Conner, key string, expiration time.Duration, ...) (_ interface{}, found bool, _ error)
• func CreateKey(prefix bool, sep string, visual string, args ...interface{}) string
• func CreateKeyStruct(strct interface{}) string
• func Hash(key string) string
• type Cacher
• type Conner
• type Logger
• type Options
• type SlowRetrieve

Functions

func Cache

func Cache(ctx context.Context, c Conner, key string, expiration time.Duration, fn SlowRetrieve, options ...Options) (_ interface{}, found bool, _ error)

Cache is used to return a cached value. If it's not available, fn will be called to obtain a value. Subsequently, fn's value will be saved into the cache.

func CreateKey

func CreateKey(prefix bool, sep string, visual string, args ...interface{}) string

CreateKey will generate a key based on the input arguments. When prefix is true, the caller's name will be used to prefix the key in an attempt to make it unique. The args can also be separated using sep. visual performs no functionality. It is used at code level to visually see how the key is structured.

func CreateKeyStruct

func CreateKeyStruct(strct interface{}) string

CreateKeyStruct generates a key by converting a struct into a JSON object.

func Hash

func Hash(key string) string

Hash returns a crc32 hashed version of key.

Types

type Cacher

type Cacher interface {
	// StorePointer sets whether a storage driver requires itemToStore to be
	// stored as a pointer or as a concrete value.
	StorePointer() bool

	// Get returns a value from the cache if the key exists.
	Get(key string) (item interface{}, found bool, err error)

	// Set sets a item into the cache for a particular key.
	Set(key string, expiration time.Duration, itemToStore interface{}) error

	// Close returns the connection back to the pool for storage drivers that utilize a pool.
	Close()

	// Forget clears the value from the cache for the particular key.
	Forget(key string) error

	// ForgetAll clears all values from the cache.
	ForgetAll() error
}

Cacher is the interface that all storage drivers must implement.

type Conner

type Conner interface {
	Conn(ctx context.Context) (Cacher, error)
}

Conner allows a storage driver to provide a connection from the pool in order to communicate with it.

type Logger

type Logger interface {
	// Log follows the same pattern as fmt.Printf( ).
	Log(format string, args ...interface{})
}

Logger provides an interface to log extra debug information. The glog package can be used or alternatively you can defined your own.

Example:

import log

type aLogger struct {}

func (l aLogger) Log(format string, args ...interface{}) {
   log.Printf(format, args...)
}

type Options

type Options struct {

	// DisableCacheUsage disables the cache.
	// It can be useful during debugging.
	DisableCacheUsage bool

	// UseFreshData will ignore content in the cache and always pull fresh data.
	// The pulled data will subsequently be saved in the cache.
	UseFreshData bool

	// Logger, when set, will log error and debug messages.
	Logger Logger

	// OnlyLogErrors, when set, will only log errors (but not debug messages).
	// For production, this should be set to true.
	OnlyLogErrors bool

	// GobRegister registers with the gob encoder the data type returned by the
	// SlowRetrieve function.
	// Some storage drivers may require this to be set.
	// Setting this to true will slightly impact concurrency performance.
	// It is usually better to set this to false, but register all structs
	// inside an init(). Otherwise you will encounter complaints from the gob package
	// if a Logger is provided.
	// See: https://golang.org/pkg/encoding/gob/#Register
	GobRegister bool
}

Options is used to change caching behavior.

type SlowRetrieve

type SlowRetrieve func(ctx context.Context) (interface{}, error)

SlowRetrieve obtains a result when the key is not found in the cache. It is usually (but not limited to) a query to a database with some additional processing of the returned data. The function must return a value that is compatible with the gob package for some storage drivers.