pnf

Documentation

Overview

Package pnf processes and then forwards messages belonging to a window. It reads the data from PBQ (which is populated by the `readloop`), calls the UDF reduce function, and then forwards to the next ISB. After a successful forward, it invokes `GC` to clean up the PBQ. Since pnf is a reducer, it mutates the watermark. The watermark after the pnf will be the end time of the window.

Index

• type ProcessAndForward
  • func NewProcessAndForward(ctx context.Context, vertexName string, pipelineName string, vr int32, ...) *ProcessAndForward
  • func (p *ProcessAndForward) Forward(ctx context.Context) error
  • func (p *ProcessAndForward) Process(ctx context.Context) error

Types

type ProcessAndForward

type ProcessAndForward struct {
	PartitionID partition.ID
	UDF         applier.ReduceApplier
	// contains filtered or unexported fields
}

ProcessAndForward reads messages from pbq, invokes udf using grpc, forwards the results to ISB, and then publishes the watermark for that partition.

func NewProcessAndForward

func NewProcessAndForward(ctx context.Context,
	vertexName string,
	pipelineName string,
	vr int32,
	partitionID partition.ID,
	udf applier.ReduceApplier,
	pbqReader pbq.Reader,
	toBuffers map[string]isb.BufferWriter,
	whereToDecider forward.ToWhichStepDecider, pw map[string]publish.Publisher) *ProcessAndForward

NewProcessAndForward will return a new ProcessAndForward instance

func (*ProcessAndForward) Forward

func (p *ProcessAndForward) Forward(ctx context.Context) error

Forward writes messages to the ISBs, publishes watermark, and invokes GC on PBQ.

func (*ProcessAndForward) Process

func (p *ProcessAndForward) Process(ctx context.Context) error

Process method reads messages from the supplied PBQ, invokes UDF to reduce the result.