mapreduce

README

go-mapreduce

Go Map/Reduce Package

Functionality

The goal of go-mapreduce is to simplify the use of the common map/reduce pattern in Go. Essentially it takes the following 2 functions:

Map: Fan out function that typically retrieves data and applies some sort of transformation.

Reduce: Aggregation function that iterates over data emitted by the Map.

TODO

• Map Function
• Reduce Function
• Report process stats
• Docs
• Tests
• Benchmarks

Documentation

Overview

Package mapreduce provides a simple abstraction for the general Map/Reduce pattern.

Example

conf := NewMapReduceConfig()

// Feed input channel
go func(in chan interface{}) {
	for i := 0; i < 100; i++ {
		in <- i
	}
	close(in)
}(conf.InChan)

var mr ExampleMapReducer

result, _ := Run(&mr, conf)

fmt.Println(result)

Output:

4950

Index

• func Run(mr MapReduce, c *Configuration) (interface{}, error)
• type Configuration
  • func NewMapReduceConfig() *Configuration
• type MapReduce

Examples

• Package

Functions

func Run

func Run(mr MapReduce, c *Configuration) (interface{}, error)

Run executes the MapReduce process.

Types

type Configuration

type Configuration struct {
	MapperCount int
	InChan      chan interface{}
	OutChan     chan interface{}
}

Configuration used by the Map Reducer.

func NewMapReduceConfig

func NewMapReduceConfig() *Configuration

NewMapReduceConfig returns a MapReduce Configuration struct with sensible defaults.

type MapReduce

type MapReduce interface {
	Map(in chan interface{}, out chan interface{})
	Reduce(in chan interface{}) interface{}
}

In order to utilize this package you must create a struct that implements the following interface.