pipeline

README

pipeline

staged stream-processing pipeline framework in go

Documentation

Index

• Variables
• type Pipeline
  • func New() *Pipeline
  • func (p *Pipeline) AddStage(name string, threads int, outChanSize int, transform transform)
  • func (p *Pipeline) Run()
  • func (p *Pipeline) SetSink(name string, threads int, commit publish)
  • func (p *Pipeline) SetSource(name string, threads int, outChanSize int, ingest ingest)

Variables

var ErrorSourceFinished = errors.New("source finished")

ErrorSourceFinished should be returned by the ingest action when the source has no input data left

Types

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline represents a stream processing pipeline

func New

func New() *Pipeline

New is the pipeline constructor

func (*Pipeline) AddStage

func (p *Pipeline) AddStage(name string, threads int, outChanSize int, transform transform)

AddStage adds a stage to the processing pipeline

func (*Pipeline) Run

func (p *Pipeline) Run()

Run runs the pipeline and blocks until done

func (*Pipeline) SetSink

func (p *Pipeline) SetSink(name string, threads int, commit publish)

SetSink sets data sink in the pipeline

func (*Pipeline) SetSource

func (p *Pipeline) SetSource(name string, threads int, outChanSize int, ingest ingest)

SetSource sets data ingestion source in the pipeline