batch

README

batch

Very simple batching API for Go.

This repo came out of a tutorial blog post.

Install

You might as well just copy the batch.go file into your own project (and the batch_test.go while you're at it for future generations) rather than adding a dependency.

But it is maintained as a Go module which you can get with:

go get github.com/pacedotdev/batch

Usage

Import it:

import (
	"github.com/pacedotdev/batch"
)

If we wanted to perform an operation in batches of ten, regardless of how many items were in the slice, we could use the batch.All function by passing in the total number of items len(items), and the batchSize. The function is a callback that gets called for each batch, with start and end indexes which you can use to re-slice the items:

items, err := getAllItemsFromRequest(r)
if err != nil {
	return errors.Wrap(err, "getAllItemsFromRequest")
}
batchSize := 10
err := batch.All(len(items), batchSize, func(start, end int) error {
	batchItems := items[start:end]
	if err := performSomeRemoteThing(ctx, batchItems); err != nil {
		return errors.Wrap(err, "performSomeRemoteThing")
	}
})
if err != nil {
	return err
}

In this example, if we got 105 items, the performSomeRemoteThing function would get called eleven times, each time with a different page of 10 items (the batchSize) except the last time, when it would be a slice of the remaining five items.

The mechanics are fairly simple, but the code is encapsulated and well tested.

Using context for cancellation

You can use a context.Context for cancellation by adding a check into the body of the BatchFunc. The following code will cancel batching if the user aborts the HTTP request.

ctx := r.Context()
items, err := getAllItemsFromRequest(r)
if err != nil {
	return errors.Wrap(err, "getAllItemsFromRequest")
}
batchSize := 10
err := batch.All(len(items), batchSize, func(start, end int) error {
	if err := ctx.Err(); err != nil {
		return err
	}
	batchItems := items[start:end]
	if err := performSomeRemoteThing(ctx, batchItems); err != nil {
		return errors.Wrap(err, "performSomeRemoteThing")
	}
})
if err != nil {
	return err
}

Documentation

Index

• Variables
• func All(count, batchSize int, eachFn BatchFunc) error
• type BatchFunc

Variables

var Abort = errors.New("done")

Abort is a sentinal error as defined by Dave Cheney (@davecheney) which indicates a batch operation should abort early.

Functions

func All

func All(count, batchSize int, eachFn BatchFunc) error

All calls eachFn for all items Returns any error from eachFn except for Abort it returns nil.

Types

type BatchFunc

type BatchFunc func(start, end int) error

BatchFunc is called for each batch. Any error will cancel the batching operation but returning Abort indicates it was deliberate, and not an error case.