mapreduce

Documentation

Index

• func FilterOutDuplicates(out chan<- Value, in <-chan KeyValue)
• func FilterOutUniques(out chan<- Value, in <-chan KeyValue)
• func Map(in <-chan Value, mapFn MapFn) <-chan KeyValue
• func Pipeline(pairs ...MapReducePair) <-chan Value
• func Reduce(in <-chan KeyValue, reduceFn ReduceFn) <-chan Value
• type KVType
  • func NewKVType(k Key, v Value) *KVType
  • func (kvt KVType) Key() KeyType
  • func (kvt KVType) Value() interface{}
• type Key
• type KeyType
  • func KeyTypeFromInt(i int) KeyType
  • func KeyTypeFromInt64(i int64) KeyType
  • func KeyTypeFromString(s string) KeyType
  • func (kt KeyType) Key() KeyType
  • func (kt KeyType) String() string
• type KeyValue
• type MapFn
• type MapReducePair
• type ReduceFn
• type Value

Functions

func FilterOutDuplicates

func FilterOutDuplicates(out chan<- Value, in <-chan KeyValue)

FilterOutDuplicates is a standard reducer that drops values with duplicate keys sending out only unique matches.

func FilterOutUniques

func FilterOutUniques(out chan<- Value, in <-chan KeyValue)

FilterOutUniques is a standard reducer that drops values with unique keys sending out the rest of the values.

func Map

func Map(in <-chan Value, mapFn MapFn) <-chan KeyValue

Map run provided Key/Value generator in a separate go-routine. It returns channel of KeyValue values.

func Pipeline

func Pipeline(pairs ...MapReducePair) <-chan Value

Pipeline builds a pipeline by chaining together provided Map/Reducer pairs. It returns a <-chan of Value.

func Reduce

func Reduce(in <-chan KeyValue, reduceFn ReduceFn) <-chan Value

Reduce takes input KeyValue channel and reduce function. It returns Value channel used by reducer for output.

Types

type KVType

type KVType struct {
	// contains filtered or unexported fields
}

func NewKVType

func NewKVType(k Key, v Value) *KVType

func (KVType) Key

func (kvt KVType) Key() KeyType

func (KVType) Value

func (kvt KVType) Value() interface{}

type Key

type Key interface {
	Key() KeyType
}

Key interface is there to make sure that the value can be used in go built-in map as a key.

type KeyType

type KeyType string

func KeyTypeFromInt

func KeyTypeFromInt(i int) KeyType

func KeyTypeFromInt64

func KeyTypeFromInt64(i int64) KeyType

func KeyTypeFromString

func KeyTypeFromString(s string) KeyType

func (KeyType) Key

func (kt KeyType) Key() KeyType

func (KeyType) String

func (kt KeyType) String() string

type KeyValue

type KeyValue interface {
	Key
	Value
}

KeyValue interface describes objects produced by Map and used by Reduce as its input stream

type MapFn

type MapFn func(out chan<- KeyValue, in <-chan Value)

MapFn provided by the client code. It is responsible to perform actual work and send it to the out channel as KeyValue tuple.

type MapReducePair

type MapReducePair struct {
	Map    MapFn
	Reduce ReduceFn
}

MapReducePair is a necessary type for pipeline builder

type ReduceFn

type ReduceFn func(out chan<- Value, in <-chan KeyValue)

ReduceFn a function provided by the client code. It expects to aggregate over the values provided by the in channel of KeyValue objects and send its result to out Value channel.

type Value

type Value interface {
	Value() interface{}
}

Value interface for objects that Map expects it its input stream