streams

README

go-streams

Go stream processing library.
Provides simple and concise DSL to build data pipelines.

Wiki
In computing, a pipeline, also known as a data pipeline,[1] is a set of data processing elements connected in series, where the output of one element is the input of the next one. The elements of a pipeline are often executed in parallel or in time-sliced fashion. Some amount of buffer storage is often inserted between elements.

Overview

Building blocks:

• Source - A Source is a set of stream processing steps that has one open output.
• Flow - A Flow is a set of stream processing steps that has one open input and one open output.
• Sink - A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.

Flow capabilities (flow package):

• Map
• FlatMap
• Filter
• PassThrough
• Split
• FanOut
• Merge
• Throttler
• SlidingWindow
• TumblingWindow

Supported Connectors:

• Go channels
• File system
• Network (TCP, UDP)
• WebSocket
• Aerospike
• Apache Kafka
• Apache Pulsar
• Redis

Examples

Could be found in the examples directory.

License

Licensed under the MIT License.

Documentation

Index

• type Flow
• type Inlet
• type Outlet
• type Sink
• type Source

Types

type Flow

type Flow interface {
	Inlet
	Outlet
	Via(Flow) Flow
	To(Sink)
}

A Flow is a set of stream processing steps that has one open input and one open output.

type Inlet

type Inlet interface {
	In() chan<- interface{}
}

Inlet is a type that exposes one open input. Implemented by the Flow and Sink.

type Outlet

type Outlet interface {
	Out() <-chan interface{}
}

Outlet is a type that exposes one open output. Implemented by the Source and Flow.

type Sink

type Sink interface {
	Inlet
}

A Sink is a set of stream processing steps that has one open input. Can be used as a Subscriber.

type Source

type Source interface {
	Outlet
	Via(Flow) Flow
}

A Source is a set of stream processing steps that has one open output.