argo-dataflow

README

Argo Dataflow

Summary

Argo Dataflow is a cloud-native platform for executing large parallel data-processing pipelines.

Each pipeline is a Kubernetes custom resource which consists of one or more steps which source and sink messages from data sources such Kafka, NATS Streaming, or HTTP services.

Each step runs 0 or more pods, and can scale horizontally using HPA or based on queue length using built-in scaling rules. Steps can be scaled-to-zero, in which case they periodically briefly scale-to-one to measure queue length so they can scale a back up

Use Cases

• Real-time "click" analytics
• Anomaly detection
• Fraud detection
• Operational (including IoT) analytics

Screenshot

Example

pip install git+https://github.com/argoproj-labs/argo-dataflow#subdirectory=dsls/python

from argo_dataflow import cron, pipeline

if __name__ == '__main__':
    (pipeline('hello')
     .namespace('argo-dataflow-system')
     .step(
        (cron('*/3 * * * * *')
         .cat('main')
         .log())
    )
     .run())

Documentation

Read in order:

Beginner:

• Quick start
• Examples
• Steps
• Sources and Sinks
• Garbage collection

Intermediate:

• Handlers
• Git usage
• Reliability
• Scaling
• Command line
• Expression syntax
• Workflow interop
• Kubectl

Advanced

• Configuration
• Features
• Metrics
• Image contract
• Reading material
• Security
• Contributing

Architecture Diagram