dataloader

README

DataLoader

This is an implementation of Facebook's DataLoader in Golang.

Install

go get -u github.com/graph-gophers/dataloader

Usage

// setup batch function - the first Context passed to the Loader's Load
// function will be provided when the batch function is called.
batchFn := func(ctx context.Context, keys dataloader.Keys) []*dataloader.Result {
  var results []*dataloader.Result
  // do some async work to get data for specified keys
  // append to this list resolved values
  return results
}

// create Loader with an in-memory cache
loader := dataloader.NewBatchedLoader(batchFn)

/**
 * Use loader
 *
 * A thunk is a function returned from a function that is a
 * closure over a value (in this case an interface value and error).
 * When called, it will block until the value is resolved.
 *
 * loader.Load() may be called multiple times for a given batch window.
 * The first context passed to Load is the object that will be passed
 * to the batch function.
 */
thunk := loader.Load(context.TODO(), dataloader.StringKey("key1")) // StringKey is a convenience method that make wraps string to implement `Key` interface
result, err := thunk()
if err != nil {
  // handle data error
}

log.Printf("value: %#v", result)

Don't need/want to use context?

You're welcome to install the v1 version of this library.

Cache

This implementation contains a very basic cache that is intended only to be used for short lived DataLoaders (i.e. DataLoaders that only exist for the life of an http request). You may use your own implementation if you want.

it also has a NoCache type that implements the cache interface but all methods are noop. If you do not wish to cache anything.

Examples

There are a few basic examples in the example folder.

Documentation

Overview

Package dataloader is an implementation of facebook's dataloader in go. See https://github.com/facebook/dataloader for more information

Index

• type BatchFunc
• type Cache
• type InMemoryCache
  • func NewCache[K comparable, V any]() *InMemoryCache[K, V]
  • func (c *InMemoryCache[K, V]) Clear()
  • func (c *InMemoryCache[K, V]) Delete(ctx context.Context, key K) bool
  • func (c *InMemoryCache[K, V]) Get(_ context.Context, key K) (Thunk[V], bool)
  • func (c *InMemoryCache[K, V]) Set(_ context.Context, key K, value Thunk[V])
• type Interface
• type Loader
  • func NewBatchedLoader[K comparable, V any](batchFn BatchFunc[K, V], opts ...Option[K, V]) *Loader[K, V]
  • func (l *Loader[K, V]) Clear(ctx context.Context, key K) Interface[K, V]
  • func (l *Loader[K, V]) ClearAll() Interface[K, V]
  • func (l *Loader[K, V]) Load(originalContext context.Context, key K) Thunk[V]
  • func (l *Loader[K, V]) LoadMany(originalContext context.Context, keys []K) ThunkMany[V]
  • func (l *Loader[K, V]) Prime(ctx context.Context, key K, value V) Interface[K, V]
• type NoCache
  • func (c *NoCache[K, V]) Clear()
  • func (c *NoCache[K, V]) Delete(context.Context, K) bool
  • func (c *NoCache[K, V]) Get(context.Context, K) (Thunk[V], bool)
  • func (c *NoCache[K, V]) Set(context.Context, K, Thunk[V])
• type NoopTracer
  • func (NoopTracer[K, V]) TraceBatch(ctx context.Context, keys []K) (context.Context, TraceBatchFinishFunc[V])
  • func (NoopTracer[K, V]) TraceLoad(ctx context.Context, key K) (context.Context, TraceLoadFinishFunc[V])
  • func (NoopTracer[K, V]) TraceLoadMany(ctx context.Context, keys []K) (context.Context, TraceLoadManyFinishFunc[V])
• type Option
  • func WithBatchCapacity[K comparable, V any](c int) Option[K, V]
  • func WithCache[K comparable, V any](c Cache[K, V]) Option[K, V]
  • func WithClearCacheOnBatch[K comparable, V any]() Option[K, V]
  • func WithInputCapacity[K comparable, V any](c int) Option[K, V]
  • func WithTracer[K comparable, V any](tracer Tracer[K, V]) Option[K, V]
  • func WithWait[K comparable, V any](d time.Duration) Option[K, V]
• type PanicErrorWrapper
  • func (p *PanicErrorWrapper) Error() string
• type Result
• type ResultMany
• type Thunk
• type ThunkMany
• type TraceBatchFinishFunc
• type TraceLoadFinishFunc
• type TraceLoadManyFinishFunc
• type Tracer

Types

type BatchFunc

type BatchFunc[K comparable, V any] func(context.Context, []K) []*Result[V]

BatchFunc is a function, which when given a slice of keys (string), returns a slice of `results`. It's important that the length of the input keys matches the length of the output results.

The keys passed to this function are guaranteed to be unique

type Cache

type Cache[K comparable, V any] interface {
	Get(context.Context, K) (Thunk[V], bool)
	Set(context.Context, K, Thunk[V])
	Delete(context.Context, K) bool
	Clear()
}

The Cache interface. If a custom cache is provided, it must implement this interface.

type InMemoryCache

type InMemoryCache[K comparable, V any] struct {
	// contains filtered or unexported fields
}

InMemoryCache is an in memory implementation of Cache interface. This simple implementation is well suited for a "per-request" dataloader (i.e. one that only lives for the life of an http request) but it's not well suited for long lived cached items.

func NewCache

func NewCache[K comparable, V any]() *InMemoryCache[K, V]

NewCache constructs a new InMemoryCache

func (*InMemoryCache[K, V]) Clear

func (c *InMemoryCache[K, V]) Clear()

Clear clears the entire cache

func (*InMemoryCache[K, V]) Delete

func (c *InMemoryCache[K, V]) Delete(ctx context.Context, key K) bool

Delete deletes item at `key` from cache

func (*InMemoryCache[K, V]) Get

func (c *InMemoryCache[K, V]) Get(_ context.Context, key K) (Thunk[V], bool)

Get gets the value at `key` if it exists, returns value (or nil) and bool indicating of value was found

func (*InMemoryCache[K, V]) Set

func (c *InMemoryCache[K, V]) Set(_ context.Context, key K, value Thunk[V])

Set sets the `value` at `key` in the cache

type Interface

type Interface[K comparable, V any] interface {
	Load(context.Context, K) Thunk[V]
	LoadMany(context.Context, []K) ThunkMany[V]
	Clear(context.Context, K) Interface[K, V]
	ClearAll() Interface[K, V]
	Prime(ctx context.Context, key K, value V) Interface[K, V]
}

Interface is a `DataLoader` Interface which defines a public API for loading data from a particular data back-end with unique keys such as the `id` column of a SQL table or document name in a MongoDB database, given a batch loading function.

Each `DataLoader` instance should contain a unique memoized cache. Use caution when used in long-lived applications or those which serve many users with different access permissions and consider creating a new instance per web request.

type Loader

type Loader[K comparable, V any] struct {
	// contains filtered or unexported fields
}

Loader implements the dataloader.Interface.

func NewBatchedLoader

func NewBatchedLoader[K comparable, V any](batchFn BatchFunc[K, V], opts ...Option[K, V]) *Loader[K, V]

NewBatchedLoader constructs a new Loader with given options.

func (*Loader[K, V]) Clear

func (l *Loader[K, V]) Clear(ctx context.Context, key K) Interface[K, V]

Clear clears the value at `key` from the cache, it it exists. Returns self for method chaining

func (*Loader[K, V]) ClearAll

func (l *Loader[K, V]) ClearAll() Interface[K, V]

ClearAll clears the entire cache. To be used when some event results in unknown invalidations. Returns self for method chaining.

func (*Loader[K, V]) Load

func (l *Loader[K, V]) Load(originalContext context.Context, key K) Thunk[V]

Load load/resolves the given key, returning a channel that will contain the value and error. The first context passed to this function within a given batch window will be provided to the registered BatchFunc.

func (*Loader[K, V]) LoadMany

func (l *Loader[K, V]) LoadMany(originalContext context.Context, keys []K) ThunkMany[V]

LoadMany loads multiple keys, returning a thunk (type: ThunkMany) that will resolve the keys passed in.

func (*Loader[K, V]) Prime

func (l *Loader[K, V]) Prime(ctx context.Context, key K, value V) Interface[K, V]

Prime adds the provided key and value to the cache. If the key already exists, no change is made. Returns self for method chaining

type NoCache

type NoCache[K comparable, V any] struct{}

NoCache implements Cache interface where all methods are noops. This is useful for when you don't want to cache items but still want to use a data loader

func (*NoCache[K, V]) Clear

func (c *NoCache[K, V]) Clear()

Clear is a NOOP

func (*NoCache[K, V]) Delete

func (c *NoCache[K, V]) Delete(context.Context, K) bool

Delete is a NOOP

func (*NoCache[K, V]) Get

func (c *NoCache[K, V]) Get(context.Context, K) (Thunk[V], bool)

Get is a NOOP

func (*NoCache[K, V]) Set

func (c *NoCache[K, V]) Set(context.Context, K, Thunk[V])

Set is a NOOP

type NoopTracer

type NoopTracer[K comparable, V any] struct{}

NoopTracer is the default (noop) tracer

func (NoopTracer[K, V]) TraceBatch

func (NoopTracer[K, V]) TraceBatch(ctx context.Context, keys []K) (context.Context, TraceBatchFinishFunc[V])

TraceBatch is a noop function

func (NoopTracer[K, V]) TraceLoad

func (NoopTracer[K, V]) TraceLoad(ctx context.Context, key K) (context.Context, TraceLoadFinishFunc[V])

TraceLoad is a noop function

func (NoopTracer[K, V]) TraceLoadMany

func (NoopTracer[K, V]) TraceLoadMany(ctx context.Context, keys []K) (context.Context, TraceLoadManyFinishFunc[V])

TraceLoadMany is a noop function

type Option

type Option[K comparable, V any] func(*Loader[K, V])

Option allows for configuration of Loader fields.

func WithBatchCapacity

func WithBatchCapacity[K comparable, V any](c int) Option[K, V]

WithBatchCapacity sets the batch capacity. Default is 0 (unbounded).

func WithCache

func WithCache[K comparable, V any](c Cache[K, V]) Option[K, V]

WithCache sets the BatchedLoader cache. Defaults to InMemoryCache if a Cache is not set.

func WithClearCacheOnBatch

func WithClearCacheOnBatch[K comparable, V any]() Option[K, V]

WithClearCacheOnBatch allows batching of items but no long term caching. It accomplishes this by clearing the cache after each batch operation.

func WithInputCapacity

func WithInputCapacity[K comparable, V any](c int) Option[K, V]

WithInputCapacity sets the input capacity. Default is 1000.

func WithTracer

func WithTracer[K comparable, V any](tracer Tracer[K, V]) Option[K, V]

WithTracer allows tracing of calls to Load and LoadMany

func WithWait

func WithWait[K comparable, V any](d time.Duration) Option[K, V]

WithWait sets the amount of time to wait before triggering a batch. Default duration is 16 milliseconds.

type PanicErrorWrapper

type PanicErrorWrapper struct {
	// contains filtered or unexported fields
}

PanicErrorWrapper wraps the error interface. This is used to check if the error is a panic error. We should not cache panic errors.

func (*PanicErrorWrapper) Error

func (p *PanicErrorWrapper) Error() string

type Result

type Result[V any] struct {
	Data  V
	Error error
}

Result is the data structure that a BatchFunc returns. It contains the resolved data, and any errors that may have occurred while fetching the data.

type ResultMany

type ResultMany[V any] struct {
	Data  []V
	Error []error
}

ResultMany is used by the LoadMany method. It contains a list of resolved data and a list of errors. The lengths of the data list and error list will match, and elements at each index correspond to each other.

type Thunk

type Thunk[V any] func() (V, error)

Thunk is a function that will block until the value (*Result) it contains is resolved. After the value it contains is resolved, this function will return the result. This function can be called many times, much like a Promise is other languages. The value will only need to be resolved once so subsequent calls will return immediately.

type ThunkMany

type ThunkMany[V any] func() ([]V, []error)

ThunkMany is much like the Thunk func type but it contains a list of results.

type TraceBatchFinishFunc

type TraceBatchFinishFunc[V any] func([]*Result[V])

type TraceLoadFinishFunc

type TraceLoadFinishFunc[V any] func(Thunk[V])

type TraceLoadManyFinishFunc

type TraceLoadManyFinishFunc[V any] func(ThunkMany[V])

type Tracer

type Tracer[K comparable, V any] interface {
	// TraceLoad will trace the calls to Load.
	TraceLoad(ctx context.Context, key K) (context.Context, TraceLoadFinishFunc[V])
	// TraceLoadMany will trace the calls to LoadMany.
	TraceLoadMany(ctx context.Context, keys []K) (context.Context, TraceLoadManyFinishFunc[V])
	// TraceBatch will trace data loader batches.
	TraceBatch(ctx context.Context, keys []K) (context.Context, TraceBatchFinishFunc[V])
}

Tracer is an interface that may be used to implement tracing.