streamflow

README

StreamFlow

StreamFlow is a high-performance distributed computing platform designed to process large-scale data streams and manage distributed services. Built with Go's concurrency model at its core, StreamFlow provides an intuitive framework for building resilient, scalable applications that harness the full potential of distributed computing.

Features

• High Performance: Leverages Go's goroutines for maximum concurrency
• Stream Processing: Real-time data stream processing with minimal latency
• Auto-Discovery: Auto peer discovery
• Security: Built-in rate limiting, authentication, and secure communication
• Auto-Scaling: Dynamic resource allocation based on workload demands
• Multi-Protocol: Support for HTTP, gRPC, and custom protocols

Architecture

StreamFlow is built on microservices architecture:

• Master-Worker Pattern: Efficient task distribution and load balancing
• Gossip Protocol: Robust peer-to-peer communication and consensus
• gRPC Framework: RPC system for service-to-service communication

Quick Start

Installation

go get -u github.com/zhenyiya/streamflow

Create Your First StreamFlow Application

1. Initialize project:

mkdir my-streamflow-app
cd my-streamflow-app
mkdir processors
touch config.json
touch main.go
cd processors
touch data_processor.go

1. Configure your cluster (config.json):
2. Create your main application (main.go):
3. Implement your data processor (processors/data_processor.go):

Launch Application

go run main.go -mode=streamflow

Application should now be running at:

• API Endpoint: http://localhost:8080/processors/DataHandler
• Dashboard: http://localhost:8080

Scale Horizontally

1. Create a second node:

cp -r my-streamflow-app my-streamflow-app-node2
cd my-streamflow-app-node2

1. Update the configuration in config.json for the second node:
2. Launch the second node:

go run main.go -mode=streamflow -port=8081

Distributed cluster is now active with automatic load balancing!

Use Cases

• Real-Time Analytics
• Financial Trading Systems
• IoT Data Processing
• Machine Learning Pipelines
• Content Delivery Networks

Documentation

Index

• func Run(vars ...*cmd.SysVars)
• func Set(key string, val ...interface{}) interface{}

Functions

func Run

func Run(vars ...*cmd.SysVars)

func Set

func Set(key string, val ...interface{}) interface{}