kstream

README

KStream - Kafka Streams for Golang

KStream is a light-weight implementation of kafka streams. It is heavily inspired by Kafka-Streams(Java) library. KStream implements features like Internal Stores, Remote Stores, Local Tables, Global Tables and it guarantees Exactly Once Processing with its de-duplicator. It can process message as micro batches or one by one depending on the throughput required. KStream can handle good amount of throughput (50,000 messages pe second with micro batch enabled) in a fault tolerable manner with a very minimal amount of latency (2 ~ 5 ms)

Project uses two external libraries sarama for consumers and producers

Stream Components

Stream Topology

Stream Topology is a set of processor nodes typically starts with a source node and ends with a sink node

Stream Builder

Stream builder is responsible for building stream topologies into kafka streams with their dependencies like changelog topics, re-partition topics, etc...

KStream

KStream is a regular kafka stream which takes an input topic as an upstream and process the record stream to another kafka topic(downstream topic) It supports several functions like transforming, merging to another stream, joins with other streams etc.

KTable

@TODO

Global KTable

Global KTable also a KTable except for is each instance of the application has its own copy of all the partitions and it will be running on a separate thread so applications dose'nt have to worry about handling failures.

Store

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, merges and removing duplicates.

Store Backend

Store is a pluggable local key-val store which is used by KTable, Global KTable and other oparations like joins, merges and removing duplicates.

RPC layer for Store

@TODO

Key discovery service

@TODO