slimgfast

README

Slimgfast

Slimgfast is a library that allows you to create a scalable, efficient, dynamic image origin server.

As an example, if you want an image to be resized to 640x480, you could hit the url at http://127.0.0.1/my-image.jpg?w=640&h=480 and slimgfast will dynamically resize your image to the correct dimensions, serve it, and cache it for later.

Slimgfast comes with an executable which supports a baseline default, but most advanced users will want to use it as a library for maximum configurability.

GoDoc: http://godoc.org/github.com/ericflo/slimgfast

Pronunciation

It's pronounced like "slimmage fast" :)

Getting Started

The easiest way to get a copy of slimfast is to use "go get":

go get github.com/ericflo/slimgfast/slimgfastd

In your code you can now import the library:

import "github.com/ericflo/slimgfast"

// For example:
fetcher := &slimgfast.ProxyFetcher{ProxyUrlPrefix: "http://i.imgur.com"}

To see what the default executable can do, try running:

slimgfastd proxy http://i.imgur.com

Now to load an image we can do:

open http://localhost:4400/EgLrnVL.jpg

With no arguments it will just proxy the original image which lives at http://i.imgur.com/EgLrnVL.jpg. With arguments it will resize it:

open http://localhost:4400/EgLrnVL.jpg\?w=300\&h=300

Using Slimgfast as a library

The steps for setting up a slimfast instance are fairly straightforward:

• Create a fetcher that will know how to read images from the upstream source
• Create a list of transformers, or potential operations that can be applied to the image (e.g. resize)
• Instantiate an app struct, which collects all the fetchers and transformers and handles the actual HTTP requests
• Spin up the groupcache library so it knows who its peers are
• Start the app and the http server

In fact, this is all that main.go is doing.

Creating your own Fetcher

Creating a Fetcher is straightforward, you only have to implement the Fetcher inteface, which means implementing the following:

Fetch(urlPath string, dest groupcache.Sink) error

Since it's really not all that much code, here's the body of the filesystem fetcher as an example:

// FilesystemFetcher fetches images from the filesystem.
type FilesystemFetcher struct {
    PathPrefix string
}

// Fetch opens and reads in the image data from the file requested by the user.
func (f *FilesystemFetcher) Fetch(urlPath string, dest groupcache.Sink) error {
    filePath := path.Clean(f.PathPrefix + urlPath)
    data, err := ioutil.ReadFile(filePath)
    if err != nil {
        return err
    }
    dest.SetBytes(data)
    return nil
}

Creating your own Transformer

Creating a Transformer is similarly straightforward to creating a Fetcher, you have to implement the Transformer interface, which has only one method:

Transform(req *ImageRequest, image image.Image) (image.Image, error)

So, it takes an image, and the request, and then returns the transformed image (or an error.) Here's the body of the resize transformer as an example:

import (
    "github.com/nfnt/resize"
    "image"
)

type TransformerResize struct{}

func (t *TransformerResize) Transform(req *ImageRequest, image image.Image) (image.Image, error) {
    resized := resize.Resize(
        uint(req.Width),
        uint(req.Height),
        image,
        resize.Lanczos3,
    )
    return resized, nil
}

You could easily write a transformer that uses ImageMagick or epeg, if you want either more power or more performance. Or you could write a filter to change the brightness, or the contrast, or turn it black and white, or any other interesting image transformation. Since you have access to the request, you can parse the querystring with whatever semantics makes sense for your needs.

I want something with commercial support

You should check out http://imgix.com/, which is a well-run startup that offers a similar but more advanced commercial image service.

Status

Status: Very, very alpha. It started as some code I'd written for work, but it ended up being more or less a complete rewrite, and this version hasn't seen any production traffic, ever. I'll remove this warning when I'm more confident in it and have run it in production.

Documentation

Overview

Package slimfast is a library that allows you to create a service which will resize, serve, and cache images.

For a full guide, visit https://www.github.com/ericflo/slimgfast

Index

• Constants
• type App
  • func NewApp(fetcher Fetcher, transformers []Transformer, counterFilename string, ...) (*App, error)
  • func (app *App) Close()
  • func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (app *App) Start()
• type Fetcher
• type ImageRequest
  • func ImageRequestFromCacheKey(cacheKey string) (*ImageRequest, error)
  • func ImageRequestFromURLString(rawUrl string) (*ImageRequest, error)
  • func (req *ImageRequest) CacheKey() (string, error)
  • func (req *ImageRequest) Size() (*Size, error)
• type ImageSource
  • func NewImageSource(fetcher Fetcher) *ImageSource
  • func NewImageSourceCustomCache(fetcher Fetcher, cacheName string, cacheMegabytes int64) *ImageSource
  • func (src *ImageSource) GetImageData(req *ImageRequest) ([]byte, error)
• type Job
• type Size
  • func SizeFromKey(key string) (Size, error)
  • func (size Size) Key() string
• type SizeCounter
  • func NewSizeCounter(filename string) (*SizeCounter, error)
  • func (counter *SizeCounter) Close()
  • func (counter *SizeCounter) CountSize(size *Size)
  • func (counter *SizeCounter) GetAllSizes() ([]Size, error)
  • func (counter *SizeCounter) GetTopSizesByCount(count uint) ([]Size, error)
  • func (counter *SizeCounter) Start(every time.Duration)
• type SizeFile
• type Transformer
• type TransformerResize
  • func (t *TransformerResize) Transform(req *ImageRequest, image image.Image) (image.Image, error)
• type WorkerGroup
  • func (wg *WorkerGroup) Close()
  • func (wg *WorkerGroup) Resize(imageSource *ImageSource, imageRequest *ImageRequest) ([]byte, error)
  • func (wg *WorkerGroup) Start()

Constants

const DEFAULT_CACHE_SIZE_MB = int64(128)

const DEFAULT_IMAGE_SOURCE_NAME = "slimgfast_image_source"

const RESIZED_IMAGE_SOURCE_NAME string = "slimgfast_resized_image_source"

Types

type App

type App struct {
	MaxWidth  int
	MaxHeight int
	// contains filtered or unexported fields
}

App ties together the fetcher, the transformers, handles the lifecycle of an image request, and does the actual HTTP serving.

func NewApp

func NewApp(
	fetcher Fetcher,
	transformers []Transformer,
	counterFilename string,
	numWorkers int,
	cacheMegabytes int64,
	maxWidth int,
	maxHeight int,
) (*App, error)

NewApp returns an App that is initialized and ready to be started.

func (*App) Close

func (app *App) Close()

Close signals to the worker group and size counter goroutines to exit.

func (*App) ServeHTTP

func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP is responsible for actually kicking off the image transformations and serving the image back to the user who requested it.

func (*App) Start

func (app *App) Start()

Start starts the application worker group and size counter goroutines.

type Fetcher

type Fetcher interface {
	Fetch(urlPath string, dest groupcache.Sink) error
}

Fetcher is the interface that is used to fetch images from some source, which could be the filesystem, a remote URL, or S3 -- but it could be from anywhere.

type ImageRequest

type ImageRequest struct {
	Url    string
	Width  int
	Height int
	Fit    string
}

ImageRequest captures information about which file the user wants to have transformed and served, and what transformations the user would like to make with it.

func ImageRequestFromCacheKey

func ImageRequestFromCacheKey(cacheKey string) (*ImageRequest, error)

ImageRequestFromCacheKey parses a cache key generated by CacheKey and constructs an ImageRequest out of it.

func ImageRequestFromURLString

func ImageRequestFromURLString(rawUrl string) (*ImageRequest, error)

ImageRequestFromURLString parses a URL string and constructs an ImageRequest out of it.

func (*ImageRequest) CacheKey

func (req *ImageRequest) CacheKey() (string, error)

CacheKey generates a cache key that encodes all of the information about this ImageRequest.

func (*ImageRequest) Size

func (req *ImageRequest) Size() (*Size, error)

Size is a convenience function for getting a *Size out of the parsed width and height ImageRequest members.

type ImageSource

type ImageSource struct {
	// contains filtered or unexported fields
}

ImageSource is an abstraction over a fetcher which caches intelligently, and serves as the primary internal interface to fetchers.

func NewImageSource

func NewImageSource(fetcher Fetcher) *ImageSource

NewImageSource initializes and returns an *ImageSource with sane default values.

func NewImageSourceCustomCache

func NewImageSourceCustomCache(fetcher Fetcher, cacheName string, cacheMegabytes int64) *ImageSource

NewImageSourceCustomCache initializes and returns an *ImageSource with a custom groupcache name and a custom cache size.

func (*ImageSource) GetImageData

func (src *ImageSource) GetImageData(req *ImageRequest) ([]byte, error)

GetImageData gets the image data the request asked for, either from cache or from the associated Fetcher.

type Job

type Job struct {
	ImageSource  ImageSource
	ImageRequest ImageRequest
	Result       chan []byte
	Error        chan error
}

Job is a single image resize job that can be sent over a channel to a worker.

type Size

type Size struct {
	Width  uint
	Height uint
}

Size is a Width and a Height

func SizeFromKey

func SizeFromKey(key string) (Size, error)

SizeFromKey takes a string of the form WIDTHxHEIGHT and parses it into a Size struct.

func (Size) Key

func (size Size) Key() string

Key takes the current Size object's Width and Height and generates a string of the form WIDTHxHEIGHT

type SizeCounter

type SizeCounter struct {
	// contains filtered or unexported fields
}

SizeCounter keeps track of how many times a certain size was requested, and persists this information to disk on a periodic basis.

func NewSizeCounter

func NewSizeCounter(filename string) (*SizeCounter, error)

NewSizeCounter initializes a *SizeCounter struct, loads in, and parses the persisted sizes.

func (*SizeCounter) Close

func (counter *SizeCounter) Close()

Close stops the SizeCounter from doing any more persistence.

func (*SizeCounter) CountSize

func (counter *SizeCounter) CountSize(size *Size)

CountSize notes the size of one request.

func (*SizeCounter) GetAllSizes

func (counter *SizeCounter) GetAllSizes() ([]Size, error)

GetAllSizes gets a list of all the sizes that we've seen.

func (*SizeCounter) GetTopSizesByCount

func (counter *SizeCounter) GetTopSizesByCount(count uint) ([]Size, error)

GetTopSizesByCount gets a list of all the sizes who have been requested at least `count` times.

func (*SizeCounter) Start

func (counter *SizeCounter) Start(every time.Duration)

Start starts the counter persisting its aggregated size stats to disk periodically.

type SizeFile

type SizeFile struct {
	Counts map[string]uint
}

SizeFile is a struct that is used to serialize the aggregated Size counts to a JSON file.

type Transformer

type Transformer interface {
	Transform(req *ImageRequest, image image.Image) (image.Image, error)
}

Transformer is the interface that any image transformation operations must adhere to. By implementing one Transform method which takes the ImageRequest and an image object and returns a new image object, we can achieve any effect we want.

type TransformerResize

type TransformerResize struct{}

TransformerResize is the primary Transformer that will resize images to the proper size.

func (*TransformerResize) Transform

func (t *TransformerResize) Transform(req *ImageRequest, image image.Image) (image.Image, error)

Transform resizes the image as requested.

type WorkerGroup

type WorkerGroup struct {
	Transformers []Transformer
	NumWorkers   int
	// contains filtered or unexported fields
}

WorkerGroup is a pool of workers, a channel, and a list of Transformers that this pool supports.

func (*WorkerGroup) Close

func (wg *WorkerGroup) Close()

Close tells the workers to quit.

func (*WorkerGroup) Resize

func (wg *WorkerGroup) Resize(imageSource *ImageSource, imageRequest *ImageRequest) ([]byte, error)

Resize enqueues one request to be run on a worker, and waits for it to respond.

func (*WorkerGroup) Start

func (wg *WorkerGroup) Start()

Start spawns workers for the worker pool and starts them up.