sprocess

README

sprocess

this library is deprecated, use Pipe instead.

sprocess is a library designed to do transformations on golang streams and http requests.

installation

go get github.com/hyperboloide/sprocess

How it works

sprocess is use to transform streams and save them somewhere. It can also do the inverse operation: get a stream and untransform it.

If you want to jump straight to the code see this example

Encoders and Decoders

Transformations are done with encoders and decoders. These are structs that transform and untransform a stream, like compress/decompress or encrypt/uncrypt.

Note that all Encoders and Decoders defines a field Name that is used for error reporting.

AES

AES encrypts and decrypts the stream.

type AES struct {
    Key          []byte // Encryption key
    Base64String string // or a base64 encoded string
    Name         string
}

You must provide either a Key or a Base64String. To encrypt in AES 256, provide a 32 bytes long key. To encrypt in AES 128, provide a 16 bytes long key

Bash

Run a command in a Bash shell. The stream is piped directly to the shell process.

type Bash struct {
    Cmd  string // Command to run ,for example "gzip" or "gzip -d" to compress and decompress.
    Name string
}

CheckSum

Make a checksum of the stream during encoding and return an error if the checksum dosen't match during decoding. The checksum is save in data with the key corresponding to the Encoder name

type CheckSum struct {
    Hash string
    Name string
}

Hash can be either "md5" or "sha1" (default if not set)

Name is mandatory

Gzip

Compress a stream.

type Gzip struct {
    Algo string // compression level
    Name string
}

Algo can have one of 3 values that correspond to the compression algorithm used :

• "best" : gzip.BestCompression
• "speed" : gzip.BestSpeed
• "default": gzip.DefaultCompression (default if not set)

Image

Transforms an image (for resize and thumbnails). Note that Image is Encoder only.

type Image struct {
    Operation     ImageOperation
    Height        uint
    Width         uint
    Interpolation string
    Output        string
    Name          string
}

ImageOperation defines the type of operation you want, it can be either:

• ImageThumbnail : to downscale an image preserving its aspect ratio to the maximum dimensions (Width, Height)
• ImageResize : to create a scaled image with new dimensions. If either Width or Height is set to 0, it will be set to an aspect ratio preserving value.

Interpolation defines the interpolation function to use (from fast to slow execution time):

• NearestNeighbor: Nearest-neighbor interpolation (default if not set)
• Bilinear: Bilinear interpolation
• Bicubic: Bicubic interpolation
• MitchellNetravali: Mitchell-Netravali interpolation
• Lanczos2: Lanczos resampling with a=2
• Lanczos3: Lanczos resampling with a=3

Output defines the output format:

• jpg (default if not set)
• png
• gif

Size

Count the number of bytes

type Size struct {
    Name string
}

The sum of bytes will be recored in data with the key Name.

Tee

Divides an input stream, this is usefull if you want to save multiple files at once. For example you may want to save a picture and at the same time create a thumbnail. Note that Tee is Encoder only.

type Tee struct {
    Encoders []Encoder
    Output   Outputer
    Name     string
}

Encoders defines the operations to be applied on the new stream. This can be empty.

Output defines where to save the output. Note that a unique Outputer should be used to avoid conflicts.

Outputs, Inputs

outputs can save a stream (to a file or an s3 bucket) and inputs read this stream back.

These may also allow for deletetion.

File

Save to the local filesystem.

type File struct {
    Prefix string // add a prefix before id to every file
	Suffix string // add a suffix after id to every file
    Dir  string // directory
    Name string
}

Streams are saved in directory. If doesn't exists, directory will be created.

S3Bucket

Save to an S3 Bucket

type S3Bucket struct {
    Prefix string // add a prefix before id to every file
	Suffix string // add a suffix after id to every file
    AccessKey string // AWS Acces Key
    SecretKey string // AWS Secret Key
    Domain    string // AWS Domain
    Bucket    string // Bucket name
    Name      string
}

AccessKey and SecretKey will be read directly from the environment if not set. To set your environment do something like this :

export AWS_ACCESS_KEY_ID="my_access_key_id"
export AWS_SECRET_ACCESS_KEY="my_secret_access_key"

Domain correspond to S3 endpoint where you created the bucket (see : http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region)

Http Handlers

sprocess also contains HTTP handlers to save, get and delete files.

An example can be found here : https://gist.github.com/fdelbos/0c1a0b47ae2cab0e971f#file-example-go

Documentation

Index

• Constants
• Variables
• func GenId() string
• type AES
  • func (c *AES) Decode(r io.Reader, w io.Writer, d *Data) error
  • func (c *AES) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (c *AES) GetName() string
  • func (c *AES) Start() error
• type Access
• type Base
• type Bash
  • func (b *Bash) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (b *Bash) GetName() string
  • func (b *Bash) Start() error
• type CheckSum
  • func (c *CheckSum) Decode(r io.Reader, w io.Writer, d *Data) error
  • func (c *CheckSum) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (c *CheckSum) GetName() string
  • func (c *CheckSum) Start() error
• type Data
  • func NewData() *Data
  • func NewDataFrom(o map[string]interface{}) *Data
  • func (d *Data) Export() map[string]interface{}
  • func (d *Data) Filter() ([]byte, error)
  • func (d *Data) Get(key string) (interface{}, error)
  • func (d *Data) Set(key string, value interface{})
  • func (d *Data) ToJson() ([]byte, error)
• type DataMap
• type Decoder
• type DecodingPipeline
• type Deleter
• type EncodeDecoder
• type Encoder
• type EncodingPipeline
  • func (ep *EncodingPipeline) GetOutputs() []string
• type File
  • func (s *File) Delete(id string, d *Data) error
  • func (s *File) GetName() string
  • func (s *File) NewReader(id string, d *Data) (io.ReadCloser, error)
  • func (s *File) NewWriter(id string, d *Data) (io.WriteCloser, error)
  • func (s *File) RemoveIfEmpty(dir string) error
  • func (s *File) Start() error
• type GoogleCloud
  • func (g *GoogleCloud) Delete(id string, d *Data) error
  • func (g *GoogleCloud) GetName() string
  • func (g *GoogleCloud) NewReader(id string, d *Data) (io.ReadCloser, error)
  • func (g *GoogleCloud) NewWriter(id string, d *Data) (io.WriteCloser, error)
  • func (g *GoogleCloud) Start() error
• type Gzip
  • func (c *Gzip) Decode(r io.Reader, w io.Writer, d *Data) error
  • func (c *Gzip) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (c *Gzip) GetName() string
  • func (c *Gzip) Start() error
• type HTTP
  • func (h *HTTP) Decode(w http.ResponseWriter, r *http.Request, dataMap map[string]interface{}) error
  • func (h *HTTP) Encode(w http.ResponseWriter, r *http.Request, id string) (map[string]interface{}, error)
  • func (h *HTTP) Remove(w http.ResponseWriter, r *http.Request, dataMap map[string]interface{}) error
• type Image
  • func (i *Image) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (i *Image) GetName() string
  • func (i *Image) Start() error
• type ImageOperation
• type Inputer
• type Null
  • func (n *Null) GetName() string
  • func (n *Null) NewWriter(id string, d *Data) (io.WriteCloser, error)
  • func (n *Null) Start() error
• type NullWriterCloser
  • func (nwc *NullWriterCloser) Close() error
  • func (nwc *NullWriterCloser) Write(p []byte) (int, error)
• type Outputer
• type Pipeline
  • func NewDecoding(decoders []Decoder, r io.ReadCloser, d *Data) *Pipeline
  • func NewEncoding(encoders []Encoder, r io.ReadCloser, d *Data) *Pipeline
  • func (p *Pipeline) Exec(w io.Writer) error
• type S3Bucket
  • func (s *S3Bucket) Delete(id string, d *Data) error
  • func (s *S3Bucket) GetName() string
  • func (s *S3Bucket) NewReader(id string, d *Data) (io.ReadCloser, error)
  • func (s *S3Bucket) NewWriter(id string, d *Data) (io.WriteCloser, error)
  • func (s *S3Bucket) Start() error
• type Service
  • func (s *Service) Decode(id string, w io.WriteCloser, data *Data) error
  • func (s *Service) Encode(id string, r io.ReadCloser, data *Data) error
• type Size
  • func (s *Size) Decode(r io.Reader, w io.Writer, d *Data) error
  • func (s *Size) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (s *Size) GetName() string
  • func (s *Size) Start() error
• type Tee
  • func (t *Tee) Encode(r io.Reader, w io.Writer, d *Data) error
  • func (t *Tee) GetName() string
  • func (t *Tee) GetOutputs() []string
  • func (t *Tee) Start() error

Constants

const GoogleCloudScope = "https://www.googleapis.com/auth/devstorage.read_write"

Variables

var NoDecodindError = errors.New("No decoding pipeline defined")

var NoEncodindError = errors.New("No encoding pipeline defined")

Functions

func GenId

func GenId() string

Types

type AES

type AES struct {
	Key          []byte
	Base64String string
	Name         string
	// contains filtered or unexported fields
}

func (*AES) Decode

func (c *AES) Decode(r io.Reader, w io.Writer, d *Data) error

func (*AES) Encode

func (c *AES) Encode(r io.Reader, w io.Writer, d *Data) error

func (*AES) GetName

func (c *AES) GetName() string

func (*AES) Start

func (c *AES) Start() error

type Access

type Access int

type Base

type Base interface {
	GetName() string
	Start() error
}

type Bash

type Bash struct {
	Cmd  string
	Name string
}

func (*Bash) Encode

func (b *Bash) Encode(r io.Reader, w io.Writer, d *Data) error

func (*Bash) GetName

func (b *Bash) GetName() string

func (*Bash) Start

func (b *Bash) Start() error

type CheckSum

type CheckSum struct {
	Hash string
	Name string
}

func (*CheckSum) Decode

func (c *CheckSum) Decode(r io.Reader, w io.Writer, d *Data) error

func (*CheckSum) Encode

func (c *CheckSum) Encode(r io.Reader, w io.Writer, d *Data) error

func (*CheckSum) GetName

func (c *CheckSum) GetName() string

func (*CheckSum) Start

func (c *CheckSum) Start() error

type Data

type Data struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func NewData

func NewData() *Data

func NewDataFrom

func NewDataFrom(o map[string]interface{}) *Data

func (*Data) Export

func (d *Data) Export() map[string]interface{}

func (*Data) Filter

func (d *Data) Filter() ([]byte, error)

func (*Data) Get

func (d *Data) Get(key string) (interface{}, error)

func (*Data) Set

func (d *Data) Set(key string, value interface{})

func (*Data) ToJson

func (d *Data) ToJson() ([]byte, error)

type DataMap

type DataMap map[string]interface{}

type Decoder

type Decoder interface {
	Base
	Decode(io.Reader, io.Writer, *Data) error
}

type DecodingPipeline

type DecodingPipeline struct {
	Decoders []Decoder
	Input    Inputer
}

type Deleter

type Deleter interface {
	Base
	Delete(string, *Data) error
}

type EncodeDecoder

type EncodeDecoder interface {
	Base
	Encode(io.Reader, io.Writer, *Data) error
	Decode(io.Reader, io.Writer, *Data) error
}

type Encoder

type Encoder interface {
	Base
	Encode(io.Reader, io.Writer, *Data) error
}

type EncodingPipeline

type EncodingPipeline struct {
	Encoders []Encoder
	Output   Outputer
}

func (*EncodingPipeline) GetOutputs

func (ep *EncodingPipeline) GetOutputs() []string

type File

type File struct {
	Prefix      string
	Suffix      string
	Dir         string
	AllowSub    bool
	RemoveEmpty bool
	Name        string
}

func (*File) Delete

func (s *File) Delete(id string, d *Data) error

func (*File) GetName

func (s *File) GetName() string

func (*File) NewReader

func (s *File) NewReader(id string, d *Data) (io.ReadCloser, error)

func (*File) NewWriter

func (s *File) NewWriter(id string, d *Data) (io.WriteCloser, error)

func (*File) RemoveIfEmpty

func (s *File) RemoveIfEmpty(dir string) error

func (*File) Start

func (s *File) Start() error

type GoogleCloud

type GoogleCloud struct {
	Prefix      string
	Suffix      string
	ProjectId   string
	Bucket      string
	Name        string
	JsonKeyPath string
	Client      *http.Client
	// contains filtered or unexported fields
}

func (*GoogleCloud) Delete

func (g *GoogleCloud) Delete(id string, d *Data) error

func (*GoogleCloud) GetName

func (g *GoogleCloud) GetName() string

func (*GoogleCloud) NewReader

func (g *GoogleCloud) NewReader(id string, d *Data) (io.ReadCloser, error)

func (*GoogleCloud) NewWriter

func (g *GoogleCloud) NewWriter(id string, d *Data) (io.WriteCloser, error)

func (*GoogleCloud) Start

func (g *GoogleCloud) Start() error

type Gzip

type Gzip struct {
	Algo string

	Name string
	// contains filtered or unexported fields
}

func (*Gzip) Decode

func (c *Gzip) Decode(r io.Reader, w io.Writer, d *Data) error

func (*Gzip) Encode

func (c *Gzip) Encode(r io.Reader, w io.Writer, d *Data) error

func (*Gzip) GetName

func (c *Gzip) GetName() string

func (*Gzip) Start

func (c *Gzip) Start() error

type HTTP

type HTTP struct {
	Encoders []Encoder
	Decoders []Decoder

	Input  Inputer
	Output Outputer
	Delete Deleter
	// contains filtered or unexported fields
}

func (*HTTP) Decode

func (h *HTTP) Decode(w http.ResponseWriter, r *http.Request, dataMap map[string]interface{}) error

func (*HTTP) Encode

func (h *HTTP) Encode(w http.ResponseWriter, r *http.Request, id string) (map[string]interface{}, error)

func (*HTTP) Remove

func (h *HTTP) Remove(w http.ResponseWriter, r *http.Request, dataMap map[string]interface{}) error

type Image

type Image struct {
	Operation     ImageOperation
	Height        uint
	Width         uint
	Interpolation string
	Output        string

	Name string
	// contains filtered or unexported fields
}

func (*Image) Encode

func (i *Image) Encode(r io.Reader, w io.Writer, d *Data) error

func (*Image) GetName

func (i *Image) GetName() string

func (*Image) Start

func (i *Image) Start() error

type ImageOperation

type ImageOperation int

const (
	ImageThumbnail ImageOperation = iota
	ImageResize
)

type Inputer

type Inputer interface {
	Base
	NewReader(string, *Data) (io.ReadCloser, error)
}

type Null

type Null struct {
	Name string
}

func (*Null) GetName

func (n *Null) GetName() string

func (*Null) NewWriter

func (n *Null) NewWriter(id string, d *Data) (io.WriteCloser, error)

func (*Null) Start

func (n *Null) Start() error

type NullWriterCloser

type NullWriterCloser struct{}

func (*NullWriterCloser) Close

func (nwc *NullWriterCloser) Close() error

func (*NullWriterCloser) Write

func (nwc *NullWriterCloser) Write(p []byte) (int, error)

type Outputer

type Outputer interface {
	Base
	NewWriter(string, *Data) (io.WriteCloser, error)
}

type Pipeline

type Pipeline struct {
	Errors []error
	// contains filtered or unexported fields
}

func NewDecoding

func NewDecoding(decoders []Decoder, r io.ReadCloser, d *Data) *Pipeline

func NewEncoding

func NewEncoding(encoders []Encoder, r io.ReadCloser, d *Data) *Pipeline

func (*Pipeline) Exec

func (p *Pipeline) Exec(w io.Writer) error

type S3Bucket

type S3Bucket struct {
	Prefix    string
	Suffix    string
	AccessKey string
	SecretKey string
	Domain    string
	Bucket    string
	Name      string
	// contains filtered or unexported fields
}

func (*S3Bucket) Delete

func (s *S3Bucket) Delete(id string, d *Data) error

func (*S3Bucket) GetName

func (s *S3Bucket) GetName() string

func (*S3Bucket) NewReader

func (s *S3Bucket) NewReader(id string, d *Data) (io.ReadCloser, error)

func (*S3Bucket) NewWriter

func (s *S3Bucket) NewWriter(id string, d *Data) (io.WriteCloser, error)

func (*S3Bucket) Start

func (s *S3Bucket) Start() error

type Service

type Service struct {
	EncodingPipe *EncodingPipeline
	DecodingPipe *DecodingPipeline
}

func (*Service) Decode

func (s *Service) Decode(id string, w io.WriteCloser, data *Data) error

func (*Service) Encode

func (s *Service) Encode(id string, r io.ReadCloser, data *Data) error

type Size

type Size struct {
	Name string
}

func (*Size) Decode

func (s *Size) Decode(r io.Reader, w io.Writer, d *Data) error

func (*Size) Encode

func (s *Size) Encode(r io.Reader, w io.Writer, d *Data) error

func (*Size) GetName

func (s *Size) GetName() string

func (*Size) Start

func (s *Size) Start() error

type Tee

type Tee struct {
	Encoders []Encoder
	Output   Outputer
	Name     string
}

func (*Tee) Encode

func (t *Tee) Encode(r io.Reader, w io.Writer, d *Data) error

func (*Tee) GetName

func (t *Tee) GetName() string

func (*Tee) GetOutputs

func (t *Tee) GetOutputs() []string

func (*Tee) Start

func (t *Tee) Start() error