simpleblob

README

Simpleblob

Simpleblob is a Go module that simplifies the storage of arbitrary data by key from Go code. It ships with the following backends:

• s3: S3 bucket storage
• fs: File storage (one file per blob)
• memory: Memory storage (for tests)

Usage

The interface implemented by the backends is:

type Interface interface {
	List(ctx context.Context, prefix string) (BlobList, error)
	Load(ctx context.Context, name string) ([]byte, error)
	Store(ctx context.Context, name string, data []byte) error
	Delete(ctx context.Context, name string) error
}

To instantiate a backend, _-import all the backends that you want to register, and call:

func GetBackend(ctx context.Context, typeName string, options map[string]any, params ...Param) (Interface, error)

An example can be found in example_test.go.

Every backend accepts a map[string]any with options and performs its own validation on the options. If you use a YAML, TOML and JSON, you could structure it like this:

type Storage struct {
	Type    string         `yaml:"type"`
	Options map[string]any `yaml:"options"` // backend specific
}

io interfaces

Reading from or writing to a blob directly can be done using the NewReader and NewWriter functions.

func NewReader(ctx context.Context, storage Interface, blobName string) (io.ReadCloser, error)
func NewWriter(ctx context.Context, storage Interface, blobName string) (io.WriteCloser, error)

The returned ReadCloser or WriteCloser is an optimized implementation if the backend being used implements the StreamReader or StreamWriter interfaces. If not, a convenience wrapper for the storage is returned.

Backend	StreamReader	StreamWriter
S3	✔	✔
Filesystem	✔	✔
Memory	✖	✖

Limitations

The interface currently does not support streaming of large blobs. In the future we may provide this by implementing fs.FS in the backend for reading, and a similar interface for writing new blobs.

API Stability

We support the last two stable Go versions, currently 1.22 and 1.23.

From a API consumer point of view, we do not plan any backward incompatible changes before a v1.0.

Documentation

Overview

Example

package main

import (
	"context"
	"fmt"

	"github.com/PowerDNS/simpleblob"
	"github.com/go-logr/logr"

	// Register the memory backend plugin
	_ "github.com/PowerDNS/simpleblob/backends/memory"
)

func main() {
	// Do not forget the:
	//     import _ "github.com/PowerDNS/simpleblob/backends/memory"
	ctx := context.Background()
	storage, err := simpleblob.GetBackend(
		ctx,
		"memory",
		map[string]interface{}{
			// add key-value options here
			"foo": "example",
		},
		simpleblob.WithLogger(logr.Discard()), // replace with a real logger
	)
	check(err)
	err = storage.Store(ctx, "example.txt", []byte("hello"))
	check(err)
	data, err := storage.Load(ctx, "example.txt")
	check(err)
	fmt.Println("data:", string(data))
	list, err := storage.List(ctx, "")
	check(err)
	for _, entry := range list {
		fmt.Println("list:", entry.Name, entry.Size)
	}
}

func check(err error) {
	if err != nil {
		panic(err)
	}
}

Output:

data: hello
list: example.txt 5

Index

• Variables
• func NewReader(ctx context.Context, st Interface, name string) (io.ReadCloser, error)
• func NewWriter(ctx context.Context, st Interface, name string) (io.WriteCloser, error)
• func RegisterBackend(typeName string, initFunc InitFunc)
• type Blob
• type BlobList
  • func (bl BlobList) Len() int
  • func (bl BlobList) Less(i, j int) bool
  • func (bl BlobList) Names() []string
  • func (bl BlobList) Size() int64
  • func (bl BlobList) Swap(i, j int)
  • func (bl BlobList) WithPrefix(prefix string) (blobs BlobList)
• type InitFunc
• type InitParams
  • func (ip InitParams) OptionsThroughYAML(dest interface{}) error
• type Interface
  • func GetBackend(ctx context.Context, typeName string, options OptionMap, params ...Param) (Interface, error)
• type OptionMap
• type Param
  • func WithLogger(log logr.Logger) Param
• type StreamReader
• type StreamWriter

Examples

• Package

Variables

var ErrClosed = fs.ErrClosed

ErrClosed implies that the Close function has already been called.

Functions

func NewReader

func NewReader(ctx context.Context, st Interface, name string) (io.ReadCloser, error)

NewReader allows reading a named blob from st. It returns an optimized io.ReadCloser if available, else a basic buffered implementation.

func NewWriter

func NewWriter(ctx context.Context, st Interface, name string) (io.WriteCloser, error)

NewWriter allows writing a named blob to st. It returns an optimized io.WriteCloser if available, else a basic buffered implementation.

func RegisterBackend

func RegisterBackend(typeName string, initFunc InitFunc)

RegisterBackend registers a new backend.

Types

type Blob

type Blob struct {
	Name string
	Size int64
}

Blob describes a single blob

type BlobList

type BlobList []Blob

BlobList is a slice of Blob structs

func (BlobList) Len

func (bl BlobList) Len() int

func (BlobList) Less

func (bl BlobList) Less(i, j int) bool

func (BlobList) Names

func (bl BlobList) Names() []string

Names returns a slice of name strings for the BlobList

func (BlobList) Size

func (bl BlobList) Size() int64

Size returns the total size of all blobs in the BlobList

func (BlobList) Swap

func (bl BlobList) Swap(i, j int)

func (BlobList) WithPrefix

func (bl BlobList) WithPrefix(prefix string) (blobs BlobList)

WithPrefix filters the BlobList to returns only Blob structs where the name starts with the given prefix.

type InitFunc

type InitFunc func(ctx context.Context, p InitParams) (Interface, error)

InitFunc is the type for the backend constructor function used to register backend.

type InitParams

type InitParams struct {
	OptionMap OptionMap // map of key-value options for this backend
	Logger    logr.Logger
}

InitParams contains the parameters for the InitFunc. This allows us to pass extra values in the future, without breaking existing backends that do not expect these.

func (InitParams) OptionsThroughYAML

func (ip InitParams) OptionsThroughYAML(dest interface{}) error

OptionsThroughYAML performs a YAML roundtrip for the OptionMap to load them into a struct with yaml tags. dest: pointer to destination struct

type Interface

type Interface interface {
	// List retrieves a BlobList with given prefix.
	List(ctx context.Context, prefix string) (BlobList, error)
	// Load brings a whole value, chosen by name, into memory.
	Load(ctx context.Context, name string) ([]byte, error)
	// Store sends value to storage for a given name.
	Store(ctx context.Context, name string, data []byte) error
	// Delete entry, identified by name, from storage. No error is returned if it does not exist.
	Delete(ctx context.Context, name string) error
}

Interface defines the interface storage plugins need to implement. This Interface MUST NOT be extended or changed without bumping the major version number, because it would break any backends in external repos. If you want to provide additional features, you can define additional optional interfaces that a backend can implement.

func GetBackend

func GetBackend(ctx context.Context, typeName string, options OptionMap, params ...Param) (Interface, error)

GetBackend creates a new backend instance of given typeName. This type must have been previously registered with RegisterBackend.

The options map contains backend dependant key-value options. Some backends take no options, others require some specific options.

Additional parameters can be passed with extra arguments, like WithLogger.

The lifetime of the context passed in must span the lifetime of the whole backend instance, not just the init time, so do not set any timeout on it!

type OptionMap

type OptionMap map[string]interface{}

OptionMap is the type for options that we pass internally to backends

type Param

type Param func(ip *InitParams)

Param is the type of extra init parameters. It is returned by calling functional params like WithLogger.

func WithLogger

func WithLogger(log logr.Logger) Param

WithLogger is a GetBackend parameter that sets the logr.Logger to use in the backends.

type StreamReader

type StreamReader interface {
	Interface
	// NewReader returns an io.ReadCloser, allowing stream reading
	// of named blob from the underlying backend.
	NewReader(ctx context.Context, name string) (io.ReadCloser, error)
}

A StreamReader is an Interface providing an optimized way to create an io.ReadCloser.

type StreamWriter

type StreamWriter interface {
	Interface
	// NewWriter returns an io.WriteCloser, allowing stream writing
	// to named blob in the underlying backend.
	NewWriter(ctx context.Context, name string) (io.WriteCloser, error)
}

A StreamWriter is an Interface providing an optimized way to create an io.WriteCloser.