golang-relyq

module
Version: v0.0.0-...-3ca95c8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 7, 2013 License: MIT

README

relyq Build Status

A relatively simple Redis-backed reliable task queue and state machine.

Its made up of four simpleq's: todo, doing, failed, and done. Tasks will never be dropped on the floor even if a processing server crashes because all operations are atomic. Tasks can be represented as any data type.

Note: relyq assumes all tasks have IDs. Thus you must pass in a pointer to an object that has an Id() method (or its element has an Id() method.)

There are a few redis clients for Go but this package uses redigo

Documentation

Documentation on godoc.org

Install

go get github.com/Rafflecopter/golang-relyq/relyq

Creation

import (
  "github.com/garyburd/redigo/redis"
  "github.com/Rafflecopter/golang-relyq/relyq"
  redisstorage "github.com/Rafflecopter/golang-relyq/storage/redis"
)

func CreateRelyQ(pool *redis.Pool) *relyq.Queue {
  cfg := &relyq.Config{
    Prefix: "my-relyq", // Required
    Delimiter: ":", // Defaults to :
    IdField: "id", // ID field for tasks
    UseDoneQueue: false, // Whether to keep list of "done" tasks (default false)
    KeepDoneTasks: false, // Whether to keep the backend storage of "done" tasks (default false)
  }

  storage := redisstorage.New(redisstorage.JSONMarshaller, pool, cfg.Prefix, cfg.Delimiter)

  return relyq.New(pool, storage, cfg)
}

func QuickCreateRelyQ(pool *redis.Pool) *relyq.Queue {
  return relyq.NewRedisJson(pool, &relyq.Config{Prefix: "my-relyq"})
}

Use

Create your task types:

type Task struct {
  relyq.StructuredTask

  IdField string
  SomethingElse *OtherStruct
}

Basic use:

q := CreateRelyQ(redisPool)

// Push a task on to Todo queue
err := q.Push(&Task{SomethingElse: &OtherStruct{}})

// Move a task from Todo to Doing
task := new(Task)
ok, err := q.Process(task) // ok=false means nothing in task

// Block and process a task
err := q.BProcess(task)

// Finish a task
err := q.Finish(task)

// Or fail it
err := q.Fail(task)

// Remove a task from the Failed queue
err := q.Remove(q.Failed, task)

// Eventually
err := q.Close()

Or use a listener:

q := CreateRelyQ(redisPool)

var example *Task
l := q.Listen(example)

go func() {
  for err := range l.Errors {
    // Do something with errors
  }
}()

go func() {
  for task := range l.Tasks {
    // Do something with tasks
    err := q.Finish(task.(*Task))
  }
}()

// Eventually
err := q.Close()

Tests

go test

Storage Options

Normal operation stores the full task description or object in the queue itself. This can be inefficient for LREM operations. Sometimes one might even want to store task descriptions in a separate datastore than redis to save memory. Custom backends have been created for this purpose.

Redis

The Redis backend stores serialized task objects in Redis. Each options object also accepts the storage_prefix field to set the prefix for where task objects are stored. Documentation

Create a storage object:

storage := redisstorage.New(marshallers.JsonMarshaller, pool, cfg.Prefix, cfg.Delimiter)

Or, for skipping the storage step, use this handy shortcut

q := relyq.NewRedisJson(pool, cfg)

TODO

  • deferred tasks
  • recurring tasks

License

See LICENSE file.

Directories

Path Synopsis
Package marshallers provides objects for marshalling tasks into bytes
Package marshallers provides objects for marshalling tasks into bytes
Package relyq provides a reliable queue backed by redis
Package relyq provides a reliable queue backed by redis
storage
redis
Package redis provides a Storage backend for relyq
Package redis provides a Storage backend for relyq

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL