querycache

package
v0.0.0-...-2d572df Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 28, 2021 License: MIT Imports: 19 Imported by: 0

README

querycache - use cache, save cash

querycache lets you save a query and access them over an HTTP API, caching the results based on a per-query "lifetime" parameter. It currently supports queries against: Postgres, Snowflake, and MySQL. And addition for more go sql compatible drives is easy!

Datasources

An Datasource is the description of the connection to a specific datasource. It has 3 parameters:

  • name - a friendly name
  • type - the driver name (e.g. postgres, mysql, snowflake)
  • options - the connection string and options.

The type and options are passed directly to sql.Open as the first and second parameter.

The following endpoints are exposed:

  • GET /datasources - List endpoint, accepts per and page query parameters
  • POST /datasources - Create endpoint, accepts json object with name, type, and options keys. (all required)
  • GET /datasources/{id} - Read endpoint, returns the JSON representation of the datasource
  • PATCH /datasources/{id} - Update endpoint, accepts json object with name, type, and options keys. (all optional)
  • DELETE /datasources/{id} - Delete endpoint, deletes the datasource

Queries

A Query represents a given query to execute.

The following endpoints are exposed:

  • GET /queries - List endpoint, accepts per and page query parameters
  • POST /queries - Create endpoint, accepts json object with query, lifetime, and datasourceId keys. (all required)
  • GET /queries/{id} - Read endpoint, returns the JSON representation of the query
  • PATCH /queries/{id} - Update endpoint, accepts json object with query, lifetime, lastRefresh, and datasourceId keys. (all optional)
  • DELETE /queries/{id} - Delete endpoint, deletes the query

Examples

The following examples are using HTTPie.

Create an datasource:

http POST http://localhost:8080/querycache/datasources type=postgres name="PG Local" options="sslmode=disable dbname=querycache"
HTTP/1.1 200 OK
Content-Length: 222
Content-Type: application/json; charset=UTF-8
Date: Sun, 31 May 2020 16:45:50 GMT

{
    "createdAt": "2020-05-31T17:45:50.595626+01:00",
    "id": "b9d444a5-720b-44f9-8ebe-6f07bb02c3e0",
    "name": "PG Local",
    "options": "sslmode=disable dbname=querycache",
    "type": "postgres",
    "updatedAt": "2020-05-31T17:45:50.595627+01:00"

Create a query:

http POST http://localhost:8080/querycache/queries datasourceId=b9d444a5-720b-44f9-8ebe-6f07bb02c3e0 query="SELECT * FROM users" lifetime="8h"         (555ms)
HTTP/1.1 200 OK
Content-Length: 290
Content-Type: application/json; charset=UTF-8
Date: Sun, 31 May 2020 16:46:42 GMT

{
    "datasourceId": "b9d444a5-720b-44f9-8ebe-6f07bb02c3e0",
    "createdAt": "2020-05-31T17:46:42.176734+01:00",
    "id": "dc927e8b-8a42-4208-8139-2c1b1af1c243",
    "lastRefresh": "2020-05-31T17:46:42.176734+01:00",
    "lifetime": "8h0m0s",
    "query": "SELECT * FROM users",
    "updatedAt": "2020-05-31T17:46:42.176734+01:00"
}

Get query results:

http GET http://localhost:8080/querycache/queries/dc927e8b-8a42-4208-8139-2c1b1af1c243/result                                                       (378ms)
HTTP/1.1 200 OK
Content-Length: 70
Content-Type: text/csv
Date: Sun, 31 May 2020 16:49:29 GMT

id,username,name,email
1,christian,Christian Gregg,christian@bissy.io

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CachedExecutor 

type CachedExecutor struct {
	Cache    QueryCache
	Executor Executor
	Store    QueryStore
	Clock    utils.Clock
}

CachedExecutor implements Executor that caches query results for the given Lifetime of a Query

func NewCachedExecutor 

func NewCachedExecutor(cache QueryCache, store QueryStore, clock utils.Clock, executor Executor) *CachedExecutor

NewCachedExecutor sets up a new CachedExecutor

func (*CachedExecutor) Execute 

func (cache *CachedExecutor) Execute(query *Query) (string, error)

Execute checks the cache for the given query cache, fallsback to the the configured executor if no results are found and stores the new results.

type Config 

type Config struct {
	QueryStore      QueryStore
	DatasourceStore DatasourceStore
	Executor        Executor
	Cache           QueryCache
	Clock           utils.Clock
}

Config contains everything external required to setup querycache

func (*Config) SetupHandlers 

func (c *Config) SetupHandlers(router *mux.Router)

SetupHandlers mounts the querycache handlers onto the given mux

type CreateDatasource 

type CreateDatasource struct {
	Name    string `json:"name"`
	Type    string `json:"type"`
	Options string `json:"options"`
}

CreateDatasource describes the required paramater to create a new Datasource

type CreateQuery 

type CreateQuery struct {
	Query        string   `json:"query"`
	Lifetime     Duration `json:"lifetime"`
	DatasourceID string   `json:"datasourceId"`
}

CreateQuery describes the required parameter to create a new Query

type Datasource 

type Datasource struct {
	ID        string    `json:"id" db:"id"`
	UserID    string    `json:"userId" db:"user_id"`
	Name      string    `json:"name"`
	Type      string    `json:"type"`
	Options   string    `json:"options"`
	CreatedAt time.Time `json:"createdAt" db:"created_at"`
	UpdatedAt time.Time `json:"updatedAt" db:"updated_at"`
}

Datasource describes a database that Queries may be related to and executed against

func (*Datasource) NewExecutor 

func (a *Datasource) NewExecutor() (Executor, error)

NewExecutor returns a new SQLExecutor configured against this Datasource Will return a TestExecutor if the datasources "Type" is "test"

type DatasourceStore 

type DatasourceStore interface {
	Get(string, string) (*Datasource, error)
	Create(string, *CreateDatasource) (*Datasource, error)
	List(string, int, int) ([]*Datasource, error)
	Delete(string, string) (*Datasource, error)
	Update(string, string, *UpdateDatasource) (*Datasource, error)
}

DatasourceStore describes a generic Store for Datasources

type Duration 

type Duration time.Duration

Duration is an alias to time.Duration to allow for defining JSON marshalling and unmarshalling

func (Duration) MarshalJSON 

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON marshals a Duration into JSON

func (*Duration) UnmarshalJSON 

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshals JSON into a Duration

type Executor 

type Executor interface {
	Execute(*Query) (string, error)
}

Executor defines the interface to execute a query

type InMemoryCache 

type InMemoryCache struct {
	Cache map[string]string
	// contains filtered or unexported fields
}

InMemoryCache is an in-memory implementation of QueryCache

func NewInMemoryCache 

func NewInMemoryCache() *InMemoryCache

NewInMemoryCache sets up a new InMemoryCache

func (*InMemoryCache) Get 

func (cache *InMemoryCache) Get(query *Query) (string, bool)

Get returns the cached results for a given query

func (*InMemoryCache) Set 

func (cache *InMemoryCache) Set(query *Query, result string) error

Set caches the results for a given query

type Query 

type Query struct {
	ID           string    `json:"id"`
	UserID       string    `json:"userId" db:"user_id"`
	Query        string    `json:"query"`
	DatasourceID string    `json:"datasourceId" db:"datasource_id"`
	Lifetime     Duration  `json:"lifetime"`
	CreatedAt    time.Time `json:"createdAt" db:"created_at"`
	UpdatedAt    time.Time `json:"updatedAt" db:"updated_at"`
	LastRefresh  time.Time `json:"lastRefresh" db:"last_refresh"`
}

Query describes an SQL query on a given datasource that should be cached for a given Lifetime value

func (*Query) Fresh 

func (query *Query) Fresh(now time.Time) bool

Fresh determines whether a query was last refreshed within Lifetime of the given time parameter

type QueryCache 

type QueryCache interface {
	Get(*Query) (string, bool)
	Set(*Query, string) error
}

QueryCache defines the interface for a cache of query results

type QueryStore 

type QueryStore interface {
	Get(string, string) (*Query, error)
	Create(string, *CreateQuery) (*Query, error)
	List(string, int, int) ([]*Query, error)
	Delete(string, string) (*Query, error)
	Update(string, string, *UpdateQuery) (*Query, error)
}

QueryStore describes a generic Store for Queries

type RedisCache 

type RedisCache struct {
	Client *redis.Client
}

RedisCache is a redis-backed implementation of QueryCache

func (*RedisCache) Get 

func (cache *RedisCache) Get(query *Query) (string, bool)

Get returns the cached results for a given query

func (*RedisCache) Set 

func (cache *RedisCache) Set(query *Query, result string) error

Set caches the results for a given query

type SQLDatasourceStore 

type SQLDatasourceStore struct {
	// contains filtered or unexported fields
}

SQLDatasourceStore describes an SQL implementation of DatasourceStore

func NewSQLDatasourceStore 

func NewSQLDatasourceStore(db *hnysqlx.DB, clock utils.Clock, generator utils.IDGenerator) *SQLDatasourceStore

NewSQLDatasourceStore retunes a new SQLDatasourceStore

func (*SQLDatasourceStore) Create 

func (s *SQLDatasourceStore) Create(userID string, ca *CreateDatasource) (*Datasource, error)

Create creates and persists a new Datasource to the Store

func (*SQLDatasourceStore) Delete 

func (s *SQLDatasourceStore) Delete(userID, id string) (*Datasource, error)

Delete removes the Datasource with given id from the Store

func (*SQLDatasourceStore) Get 

func (s *SQLDatasourceStore) Get(userID, id string) (*Datasource, error)

Get returns the Datasource with associated id from the store

func (*SQLDatasourceStore) List 

func (s *SQLDatasourceStore) List(userID string, page, per int) ([]*Datasource, error)

List returns the requests Datasources from the Store, ordered by createdAt

func (*SQLDatasourceStore) Update 

func (s *SQLDatasourceStore) Update(userID, id string, ua *UpdateDatasource) (*Datasource, error)

Update updates the Datasource with associated id from the store

type SQLExecutor 

type SQLExecutor struct {
	// contains filtered or unexported fields
}

SQLExecutor implements Executor against an *sql.DB

func NewSQLExecutor 

func NewSQLExecutor(driver, conn string) (*SQLExecutor, error)

NewSQLExecutor builds a new SQLExecutor, parameters are passed to sql.Open

func (*SQLExecutor) Execute 

func (sql *SQLExecutor) Execute(query *Query) (string, error)

Execute runs the query against the configured database returns the results as a CSV string

type SQLQueryStore 

type SQLQueryStore struct {
	// contains filtered or unexported fields
}

SQLQueryStore defines an SQL implementation of a QueryStore

func NewSQLQueryStore 

func NewSQLQueryStore(db *hnysqlx.DB, clock utils.Clock, generator utils.IDGenerator) *SQLQueryStore

NewSQLQueryStore builds a new SQLQueryStore

func (*SQLQueryStore) Create 

func (s *SQLQueryStore) Create(userID string, ca *CreateQuery) (*Query, error)

Create creates and persist to memory a Query from a CreateQuery struct

func (*SQLQueryStore) Delete 

func (s *SQLQueryStore) Delete(userID, id string) (*Query, error)

Delete removes the Query with associated id from the store

func (*SQLQueryStore) Get 

func (s *SQLQueryStore) Get(userID, id string) (*Query, error)

Get returns the Query with associated id from the store

func (*SQLQueryStore) List 

func (s *SQLQueryStore) List(userID string, page, per int) ([]*Query, error)

List returns the requests Queries from the Store, ordered by createdAt

func (*SQLQueryStore) Update 

func (s *SQLQueryStore) Update(userID, id string, uq *UpdateQuery) (*Query, error)

Update updates the Query with associated id from the store

type TestExecutor 

type TestExecutor struct{}

TestExecutor implements an Executor that echoes the passed query

func (*TestExecutor) Execute 

func (t *TestExecutor) Execute(query *Query) (string, error)

Execute echoes the given query

type UpdateDatasource 

type UpdateDatasource struct {
	Name    *string `json:"name"`
	Type    *string `json:"type"`
	Options *string `json:"options"`
}

UpdateDatasource describes the paramater which may be updated on a Datasource

type UpdateQuery 

type UpdateQuery struct {
	Lifetime    *Duration `json:"lifetime"`
	LastRefresh time.Time `json:"lastRefresh"`
}

UpdateQuery describes the paramater which may be updated on a Query

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.