pipeline

Documentation

Overview

Package pipeline contains structures that implement both the Producer and Consumer interfaces. They can be used as extra pipeline components for various utilities.

Index

• type Config
  • func NewConfig() Config
• type Pool
  • func NewPool(threads int, log log.Modular, msgProcessors ...iprocessor.V1) (*Pool, error)
  • func (p *Pool) CloseAsync()
  • func (p *Pool) Consume(msgs <-chan message.Transaction) error
  • func (p *Pool) TransactionChan() <-chan message.Transaction
  • func (p *Pool) WaitForClose(timeout time.Duration) error
• type Processor
  • func NewProcessor(msgProcessors ...iprocessor.V1) *Processor
  • func (p *Processor) CloseAsync()
  • func (p *Processor) Consume(msgs <-chan message.Transaction) error
  • func (p *Processor) TransactionChan() <-chan message.Transaction
  • func (p *Processor) WaitForClose(timeout time.Duration) error
• type Type
  • func New(conf Config, mgr bundle.NewManagement) (Type, error)

Types

type Config

type Config struct {
	Threads    int                `json:"threads" yaml:"threads"`
	Processors []processor.Config `json:"processors" yaml:"processors"`
}

Config is a configuration struct for creating parallel processing pipelines. The number of resuling parallel processing pipelines will match the number of threads specified. Processors are executed on each message in the order that they are written.

In order to fully utilise each processing thread you must either have a number of parallel inputs that matches or surpasses the number of pipeline threads, or use a memory buffer.

func NewConfig

func NewConfig() Config

NewConfig returns a configuration struct fully populated with default values.

type Pool

type Pool struct {
	// contains filtered or unexported fields
}

Pool is a pool of pipelines. Each pipeline reads from a shared transaction channel. Inputs remain coupled to their outputs as they propagate the response channel in the transaction.

func NewPool

func NewPool(threads int, log log.Modular, msgProcessors ...iprocessor.V1) (*Pool, error)

NewPool creates a new processing pool.

func (*Pool) CloseAsync

func (p *Pool) CloseAsync()

CloseAsync shuts down the pipeline and stops processing messages.

func (*Pool) Consume

func (p *Pool) Consume(msgs <-chan message.Transaction) error

Consume assigns a messages channel for the pipeline to read.

func (*Pool) TransactionChan

func (p *Pool) TransactionChan() <-chan message.Transaction

TransactionChan returns the channel used for consuming messages from this pipeline.

func (*Pool) WaitForClose

func (p *Pool) WaitForClose(timeout time.Duration) error

WaitForClose - Blocks until the StackBuffer output has closed down.

type Processor

type Processor struct {
	// contains filtered or unexported fields
}

Processor is a pipeline that supports both Consumer and Producer interfaces. The processor will read from a source, perform some processing, and then either propagate a new message or drop it.

func NewProcessor

func NewProcessor(msgProcessors ...iprocessor.V1) *Processor

NewProcessor returns a new message processing pipeline.

func (*Processor) CloseAsync

func (p *Processor) CloseAsync()

CloseAsync shuts down the pipeline and stops processing messages.

func (*Processor) Consume

func (p *Processor) Consume(msgs <-chan message.Transaction) error

Consume assigns a messages channel for the pipeline to read.

func (*Processor) TransactionChan

func (p *Processor) TransactionChan() <-chan message.Transaction

TransactionChan returns the channel used for consuming messages from this pipeline.

func (*Processor) WaitForClose

func (p *Processor) WaitForClose(timeout time.Duration) error

WaitForClose blocks until the StackBuffer output has closed down.

type Type

type Type interface {
	iprocessor.Pipeline
}

Type is implemented by all pipeline implementations.

func New

func New(conf Config, mgr bundle.NewManagement) (Type, error)

New creates an input type based on an input configuration.