mapreduce

Documentation

Overview

Package mapreduce implements the Map/Reduce algorithm for the processing and aggregation mass data.

A type implementing the MapReducer interface has to be implemented and passed to the MapReduce() function. The type is responsible for the input, the mapping, the reducing and the consuming while the package provides the runtime environment for it.

Index

• func MapReduce(mr MapReducer) error
• type Identifiable
• type IdentifiableChan
  • func (c IdentifiableChan) Close()
• type MapReducer

Functions

func MapReduce

func MapReduce(mr MapReducer) error

MapReduce applies a map and a reduce function to keys and values in parallel.

Types

type Identifiable

type Identifiable interface {
	// ID returns the identifier for the mapping.
	ID() string
}

Identifiable has to be implemented by the data handled by map/reduce.

type IdentifiableChan

type IdentifiableChan chan Identifiable

IdentifiableChan is a channel for the transfer of identifiable data.

func (IdentifiableChan) Close

func (c IdentifiableChan) Close()

Close the channel for identifiable data.

type MapReducer

type MapReducer interface {
	// Input has to return the input channel for the
	// date to process.
	Input() IdentifiableChan

	// Map maps a key/value pair to another one and emits it.
	Map(in Identifiable, emit IdentifiableChan)

	// Reduce reduces the values delivered via the input
	// channel to the emit channel.
	Reduce(in, emit IdentifiableChan)

	// Consume allows the MapReducer to consume the
	// processed data.
	Consume(in IdentifiableChan) error
}

MapReducer has to be implemented to control the map/reducing.