xmapreduce

Documentation

Index

• func Map(ctx context.Context, generateFunc GenerateFunc, mapFunc MapFunc, ...) <-chan interface{}
• func MapStream(ctx context.Context, generateFunc GenerateFunc, mapFunc MapFunc, ...) *xstream.Stream
• type GenerateFunc
• type MapFunc
• type Option
  • func WithWorkerSize(workerSize int) Option
• type ReducerFunc

Functions

func Map

func Map(ctx context.Context, generateFunc GenerateFunc, mapFunc MapFunc, opts ...Option) <-chan interface{}

Map maps all elements generated from given generate func, and returns an output channel.

func MapStream

func MapStream(ctx context.Context, generateFunc GenerateFunc, mapFunc MapFunc, opts ...Option) *xstream.Stream

MapStream maps all elements generated from given generate func, and returns a xstream.Stream.

Types

type GenerateFunc

type GenerateFunc func(source chan<- interface{})

GenerateFunc is used to let callers send elements into source.

type MapFunc

type MapFunc func(item interface{}, writer xbarrier.Writer)

MapFunc is used to do element processing and write the output to writer.

type Option

type Option func(opts *options)

Option defines the method to customize the mapreduce.

func WithWorkerSize

func WithWorkerSize(workerSize int) Option

WithWorkerSize customizes a mapreduce processing with given workers.

type ReducerFunc

type ReducerFunc func(pipe <-chan interface{}, writer xbarrier.Writer, cancel func(error))

ReducerFunc is used to reduce all the mapping output and write to writer, use cancel func to cancel the processing.