mapreduce

README

mapreduce

English | 简体中文

Why we have this repo?

mapreduce is part of go-zero, but a few people asked if mapreduce can be used separately. But I recommend you to use go-zero for many more features.

Design ideas

Let's try to put ourselves in the author's shoes and sort out the possible business scenarios for the concurrency tool:

1. querying product details: supporting concurrent calls to multiple services to combine product attributes, and supporting call errors that can be ended immediately.
2. automatic recommendation of user card coupons on product details page: support concurrently verifying card coupons, automatically rejecting them if they fail, and returning all of them.

The above is actually processing the input data and finally outputting the cleaned data. There is a very classic asynchronous pattern for data processing: the producer-consumer pattern. So we can abstract the life cycle of data batch processing, which can be roughly divided into three phases.

1. data production generate
2. data processing mapper
3. data aggregation reducer

Data producing is an indispensable stage, data processing and data aggregation are optional stages, data producing and processing support concurrent calls, data aggregation is basically a pure memory operation, so a single concurrent process can do it.

Since different stages of data processing are performed by different goroutines, it is natural to consider the use of channel to achieve communication between goroutines.

How can I terminate the process at any time?

It's simple, just listen to a global end channel or the given context in the goroutine.

A simple example

Calculate the sum of squares, simulating the concurrency.

package main

import (
    "fmt"
    "log"

    "github.com/kevwan/mapreduce"
)

func main() {
    val, err := mapreduce.MapReduce(func(source chan<- interface{}) {
        // generator
        for i := 0; i < 10; i++ {
            source <- i
        }
    }, func(item interface{}, writer mapreduce.Writer, cancel func(error)) {
        // mapper
        i := item.(int)
        writer.Write(i * i)
    }, func(pipe <-chan interface{}, writer mapreduce.Writer, cancel func(error)) {
        // reducer
        var sum int
        for i := range pipe {
            sum += i.(int)
        }
        writer.Write(sum)
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("result:", val)
}

References

go-zero: https://github.com/zeromicro/go-zero

Give a Star! ⭐

If you like or are using this project to learn or start your solution, please give it a star. Thanks!

Documentation

Index

• Variables
• func Finish(fns ...func() error) error
• func FinishVoid(fns ...func())
• func Map(generate GenerateFunc, mapper MapFunc, opts ...Option) chan interface{}
• func MapReduce(generate GenerateFunc, mapper MapperFunc, reducer ReducerFunc, opts ...Option) (interface{}, error)
• func MapReduceVoid(generate GenerateFunc, mapper MapperFunc, reducer VoidReducerFunc, ...) error
• func MapReduceWithSource(source <-chan interface{}, mapper MapperFunc, reducer ReducerFunc, ...) (interface{}, error)
• func MapVoid(generate GenerateFunc, mapper VoidMapFunc, opts ...Option)
• type AtomicError
  • func (ae *AtomicError) Load() error
  • func (ae *AtomicError) Set(err error)
• type DoneChan
  • func NewDoneChan() *DoneChan
  • func (dc *DoneChan) Close()
  • func (dc *DoneChan) Done() chan struct{}
• type GenerateFunc
• type MapFunc
• type MapperFunc
• type Option
  • func WithContext(ctx context.Context) Option
  • func WithWorkers(workers int) Option
• type ReducerFunc
• type VoidMapFunc
• type VoidReducerFunc
• type Writer

Variables

var (
	// ErrCancelWithNil is an error that mapreduce was cancelled with nil.
	ErrCancelWithNil = errors.New("mapreduce cancelled with nil")
	// ErrReduceNoOutput is an error that reduce did not output a value.
	ErrReduceNoOutput = errors.New("reduce not writing value")
)

Functions

func Finish

func Finish(fns ...func() error) error

Finish runs fns parallelly, cancelled on any error.

func FinishVoid

func FinishVoid(fns ...func())

FinishVoid runs fns parallelly.

func Map

func Map(generate GenerateFunc, mapper MapFunc, opts ...Option) chan interface{}

Map maps all elements generated from given generate func, and returns an output channel.

func MapReduce

func MapReduce(generate GenerateFunc, mapper MapperFunc, reducer ReducerFunc,
	opts ...Option) (interface{}, error)

MapReduce maps all elements generated from given generate func, and reduces the output elements with given reducer.

func MapReduceVoid

func MapReduceVoid(generate GenerateFunc, mapper MapperFunc, reducer VoidReducerFunc, opts ...Option) error

MapReduceVoid maps all elements generated from given generate, and reduce the output elements with given reducer.

func MapReduceWithSource

func MapReduceWithSource(source <-chan interface{}, mapper MapperFunc, reducer ReducerFunc,
	opts ...Option) (interface{}, error)

MapReduceWithSource maps all elements from source, and reduce the output elements with given reducer.

func MapVoid

func MapVoid(generate GenerateFunc, mapper VoidMapFunc, opts ...Option)

MapVoid maps all elements from given generate but no output.

Types

type AtomicError

type AtomicError struct {
	// contains filtered or unexported fields
}

AtomicError defines an atomic error.

func (*AtomicError) Load

func (ae *AtomicError) Load() error

Load returns the error.

func (*AtomicError) Set

func (ae *AtomicError) Set(err error)

Set sets the error.

type DoneChan

type DoneChan struct {
	// contains filtered or unexported fields
}

A DoneChan is used as a channel that can be closed multiple times and wait for done.

func NewDoneChan

func NewDoneChan() *DoneChan

NewDoneChan returns a DoneChan.

func (*DoneChan) Close

func (dc *DoneChan) Close()

Close closes dc, it's safe to close more than once.

func (*DoneChan) Done

func (dc *DoneChan) Done() chan struct{}

Done returns a channel that can be notified on dc closed.

type GenerateFunc

type GenerateFunc func(source chan<- interface{})

GenerateFunc is used to let callers send elements into source.

type MapFunc

type MapFunc func(item interface{}, writer Writer)

MapFunc is used to do element processing and write the output to writer.

type MapperFunc

type MapperFunc func(item interface{}, writer Writer, cancel func(error))

MapperFunc is used to do element processing and write the output to writer, use cancel func to cancel the processing.

type Option

type Option func(opts *mapReduceOptions)

Option defines the method to customize the mapreduce.

func WithContext

func WithContext(ctx context.Context) Option

WithContext customizes a mapreduce processing accepts a given ctx.

func WithWorkers

func WithWorkers(workers int) Option

WithWorkers customizes a mapreduce processing with given workers.

type ReducerFunc

type ReducerFunc func(pipe <-chan interface{}, writer Writer, cancel func(error))

ReducerFunc is used to reduce all the mapping output and write to writer, use cancel func to cancel the processing.

type VoidMapFunc

type VoidMapFunc func(item interface{})

VoidMapFunc is used to do element processing, but no output.

type VoidReducerFunc

type VoidReducerFunc func(pipe <-chan interface{}, cancel func(error))

VoidReducerFunc is used to reduce all the mapping output, but no output. Use cancel func to cancel the processing.

type Writer

type Writer interface {
	Write(v interface{})
}

Writer interface wraps Write method.