jobrunner

README

JobRunner

JobRunner is framework for performing work asynchronously, outside of the request flow. It comes with cron to schedule and queue job functions for processing at specified time.

It includes a live monitoring of current schedule and state of active jobs that can be outputed as JSON or Html template.

Install

go get github.com/bamzi/jobrunner

Setup

package main

import "github.com/bamzi/jobrunner"

func main() {
    jobrunner.Start() // optional: jobrunner.Start(pool int, concurrent int) (10, 1)
    jobrunner.Schedule("@every 5s", ReminderEmails{})
}

// Job Specific Functions
type ReminderEmails struct {
    // filtered
}

// ReminderEmails.Run() will get triggered automatically.
func (e ReminderEmails) Run() {
    // Queries the DB
    // Sends some email
    fmt.Printf("Every 5 sec send reminder emails \n")
}

Live Monitoring

// Example of GIN micro framework
func main() {
    routes := gin.Default()

    // Resource to return the JSON data
    routes.GET("/jobrunner/json", JobJson)

    // Load template file location relative to the current working directory
    routes.LoadHTMLGlob("../github.com/bamzi/jobrunner/views/Status.html")

    // Returns html page at given endpoint based on the loaded
    // template from above
    routes.GET("/jobrunner/html", JobHtml)

    routes.Run(":8080")
}

func JobJson(c *gin.Context) {
    // returns a map[string]interface{} that can be marshalled as JSON
    c.JSON(200, jobrunner.StatusJson())
}

func JobHtml(c *gin.Context) {
    // Returns the template data pre-parsed
    c.HTML(200, "", jobrunner.StatusPage())

}

WHY?

To reduce our http response latency by 200+%

JobRunner was created to help us process functions unrelated to response without any delays to the http response. GoRoutines would timeout because response has already been processed, closing the instance window all together.

Instead of creating a separate independent app, we wanted to save time and manageability of our current app by coupling-in the job processing. We did not want to have micro services. It's premature optimization.

If you have a web app or api service backend and want a job processing framework built into your app then JobRunner is for you. Once you hit mass growth and need to scale, you can simply decouple you JobRunners into a dedicated app.

Use cases

Here are some examples of what we use JobRunner for:

• Send emails to new users after signup
• Sending push notification or emails based on specifics
• ReMarketing Engine - send invites, reminder emails, etc ...
• Clean DB, data or AMZ S3
• Sending Server stats to monitoring apps
• Send data stats at daily or weekly intervals

Supported Featured

All jobs are processed outside of the request flow

• Now: process a job immediately
• In: processing a job one time, after a given time
• Every: process a recurring job after every given time gap
• Schedule: process a job (recurring or otherwise) at a given time

Compatibility

JobRunner is designed to be framework agnostic. So it will work with pure Go apps as well as any framework written in Go Language.

Verified Supported Frameworks

• Gin
• Echo
• Martini
• Beego
• Revel (JobRunner is modified version of revel's Jobs module)
• ...

Examples & recipes are coming soon

Basics

    jobrunner.Schedule("* */5 * * * *", DoSomething{}) // every 5min do something
    jobrunner.Schedule("@every 1h30m10s", ReminderEmails{})
    jobrunner.Schedule("@midnight", DataStats{}) // every midnight do this..
    jobrunner.Every(16*time.Minute, CleanS3{}) // evey 16 min clean...
    jobrunner.In(10*time.Second, WelcomeEmail{}) // one time job. starts after 10sec
    jobrunner.Now(NowDo{}) // do the job as soon as it's triggered

More Detailed CRON Specs

Contribute

Use issues for everything

• Report problems
• Discuss before sending pull request
• Suggest new features
• Improve/fix documentation

Credits

• revel jobs module - Origin of JobRunner
• robfig cron v3 - github.com/robfig/cron/v3
• contributors

Author

Bam Azizi Github: @bamzi Twitter: @bamazizi

License

MIT

Documentation

Overview

A job runner for executing scheduled or ad-hoc tasks asynchronously from HTTP requests.

It adds a couple of features on top of the Robfig cron package:

1. Protection against job panics. (They print to ERROR instead of take down the process)
2. (Optional) Limit on the number of jobs that may run simulatenously, to limit resource consumption.
3. (Optional) Protection against multiple instances of a single job running concurrently. If one execution runs into the next, the next will be queued.
4. Cron expressions may be defined in app.conf and are reusable across jobs.
5. Job status reporting. [WIP]

Index

• Constants
• Variables
• func Entries() []cron.Entry
• func Every(duration time.Duration, job cron.Job)
• func In(duration time.Duration, job cron.Job)
• func Now(job cron.Job)
• func Remove(id cron.EntryID)
• func Schedule(spec string, job cron.Job) error
• func Start(v ...int)
• func StatusJson() map[string]interface{}
• func Stop()
• type Func
  • func (r Func) Run()
• type Job
  • func AddJob(job cron.Job) *Job
  • func New(job cron.Job) *Job
  • func (j *Job) Run()
  • func (j *Job) StatusUpdate() string
• type StatusData
  • func StatusPage() []StatusData

Constants

const DEFAULT_JOB_POOL_SIZE = 10

const UNNAMED = "(unnamed)"

Variables

var (
	// Singleton instance of the underlying job scheduler.
	MainCron *cron.Cron
)

Functions

func Entries

func Entries() []cron.Entry

Return detailed list of currently running recurring jobs to remove an entry, first retrieve the ID of entry

func Every

func Every(duration time.Duration, job cron.Job)

Run the given job at a fixed interval. The interval provided is the time between the job ending and the job being run again. The time that the job takes to run is not included in the interval.

func In

func In(duration time.Duration, job cron.Job)

Run the given job once, after the given delay.

func Now

func Now(job cron.Job)

Run the given job right now.

func Remove

func Remove(id cron.EntryID)

Remove a specific job from running Get EntryID from the list job entries jobrunner.Entries() If job is in the middle of running, once the process is finished it will be removed

func Schedule

func Schedule(spec string, job cron.Job) error

func Start

func Start(v ...int)

func StatusJson

func StatusJson() map[string]interface{}

func Stop

func Stop()

Stop ALL active jobs from running at the next scheduled time

Types

type Func

type Func func()

Callers can use jobs.Func to wrap a raw func. (Copying the type to this package makes it more visible)

For example:

jobrunner.Schedule("cron.frequent", jobs.Func(myFunc))

func (Func) Run

func (r Func) Run()

type Job

type Job struct {
	Name string

	Status  string
	Latency string
	// contains filtered or unexported fields
}

func AddJob

func AddJob(job cron.Job) *Job

func New

func New(job cron.Job) *Job

func (*Job) Run

func (j *Job) Run()

func (*Job) StatusUpdate

func (j *Job) StatusUpdate() string

type StatusData

type StatusData struct {
	Id        cron.EntryID
	JobRunner *Job
	Next      time.Time
	Prev      time.Time
}

func StatusPage

func StatusPage() []StatusData