README
ΒΆ
English | ζ₯ζ¬θͺ
π Surfin - Batch framework
A lightweight batch framework for Go, inspired by JSR-352.
Surfin - Batch framework is being developed with robustness, scalability, and operational ease as top priorities. With declarative job definitions (JSL) and a clean architecture, it efficiently and reliably executes complex data processing tasks.
π Why Choose Surfin?
- π Go Performance: Leverages Go's native concurrency and performance to the fullest.
- ποΈ Convention over Configuration (CoC): Eliminates boilerplate code through Convention over Configuration (CoC), ensuring robustness with minimal implementation.
- β¨ Observable: Integrates Prometheus/OpenTelemetry at its core, enabling distributed tracing and metrics collection simply by following conventions.
- π οΈ Flexible: Supports dynamic DB routing and Uber Fx (DI) for flexible adaptation to complex infrastructures and custom requirements.
- π Robust: Provides fault tolerance features through persistent metadata, optimistic locking, and fine-grained error handling (retry/skip).
- π― JSR-352 Compliant: Implements a batch processing model inspired by the industry standard (JSR-352).
- π Scalable: Supports large-scale datasets through distributed execution via remote worker integration and abstraction of Partitioning.
π οΈ Key Features
βοΈ Core Features and Flow Control
- Declarative Job Definition (JSL): Define jobs, steps, components, and transitions declaratively using YAML-based JSL.
- Chunk/Tasklet Model: Supports chunk-oriented processing with
ItemReader,ItemProcessor,ItemWriter, andTaskletfor single-task execution. - Advanced Flow Control: Build complex job flows with conditional branching (
Decision), parallel execution (Split), and flexible transition rules (Transition). - Listener Model: Custom logic can be injected into Job, Step, Chunk, and Item lifecycle events.
π‘οΈ Robustness and Metadata Management
- Restartability: Failed jobs can be accurately restarted from their interruption point using
JobRepositoryandExecutionContext. - Fault Tolerance: Supports item-level retry and skip policies. Automatically applies chunk-splitting skip logic for write errors.
- Optimistic Locking: Implements optimistic locking at the metadata repository layer to ensure data consistency in distributed environments.
- Sensitive Information Masking: Automatically masks sensitive information when
JobParametersare persisted and logged.
β¨ Operational Ease and Observability
- OpenTelemetry/Prometheus Integration: Integrates distributed tracing (automatic Job/Step Span generation) and metrics collection (MetricRecorder) into the framework's core. Ensures reliable monitoring even for short-lived batches and provides a foundation for integration with Remote Schedulers.
- Fine-grained Error Policy: Supports declarative retry/skip strategies based on error characteristics (Transient, Skippable).
π Scalability and Infrastructure
- Dynamic DI: Adopts Uber Fx for dependency injection, enabling dynamic component construction and lifecycle management.
- Dynamic Data Source Routing: Supports dynamic switching between multiple data sources (Postgres, MySQL, etc.) based on execution context.
- Distributed Execution Abstraction: Abstraction of
PartitionerandStepExecutorsupports delegation of execution from local to remote workers (e.g., Kubernetes Job).
π Getting Started
Refer to the following guide to build a minimal application using Surfin.
π tutorial - "Hello, Wold!"
π Documentation & Usage
For more detailed features, configurations, and building complex job flows, refer to the following documentation.
- Architecture and Design Principles: Refer to detailed information on the framework's architecture, design principles, and project structure.
- Guide to Creating Surfin Batch Applications: A complete guide to JSL, custom components, listeners, and flow control.
- Implementation Roadmap: The framework's design goals and progress.
π Support
If you have any questions or encounter issues, please use GitHub Issues.
- GitHub Issues: Report bugs or request features
Directories
ΒΆ
| Path | Synopsis |
|---|---|
|
example
|
|
|
hello-world/cmd/hello-world
command
Package main provides the main application entry point and sets up the Fx dependency injection graph.
|
Package main provides the main application entry point and sets up the Fx dependency injection graph. |
|
hello-world/internal/app
Package app provides application-level components and utilities for the hello-world example.
|
Package app provides application-level components and utilities for the hello-world example. |
|
hello-world/internal/app/job
Package job provides the implementation of the "helloWorldJob" batch job.
|
Package job provides the implementation of the "helloWorldJob" batch job. |
|
hello-world/internal/app/runner
Package runner provides implementations for job execution runners.
|
Package runner provides implementations for job execution runners. |
|
hello-world/internal/step
Package step provides implementations for various batch steps, including tasklets.
|
Package step provides implementations for various batch steps, including tasklets. |
|
weather/cmd/weather
command
|
|
|
weather/internal/app
Package app provides the main application module for the weather batch example, setting up core dependencies.
|
Package app provides the main application module for the weather batch example, setting up core dependencies. |
|
weather/internal/component/tasklet
Package tasklet provides Fx modules for application-specific tasklets.
|
Package tasklet provides Fx modules for application-specific tasklets. |
|
weather/internal/repository
Package repository provides interfaces and implementations for weather data persistence.
|
Package repository provides interfaces and implementations for weather data persistence. |
|
weather/internal/step/writer
Package writer provides the Fx module for the WeatherItemWriter component.
|
Package writer provides the Fx module for the WeatherItemWriter component. |
|
pkg
|
|
|
batch/adapter/database/dummy
Package dummy provides dummy implementations for database-related interfaces.
|
Package dummy provides dummy implementations for database-related interfaces. |
|
batch/adapter/database/gorm/mysql
Package mysql provides a GORM DBProvider implementation for MySQL databases.
|
Package mysql provides a GORM DBProvider implementation for MySQL databases. |
|
batch/adapter/database/gorm/postgres
Package postgres provides a GORM DBProvider implementation for PostgreSQL and Redshift databases.
|
Package postgres provides a GORM DBProvider implementation for PostgreSQL and Redshift databases. |
|
batch/adapter/database/gorm/sqlite
Package sqlite provides a GORM DBProvider implementation for SQLite databases.
|
Package sqlite provides a GORM DBProvider implementation for SQLite databases. |
|
batch/adapter/storage/local
Package local provides a local file system implementation of the storage adapter interfaces.
|
Package local provides a local file system implementation of the storage adapter interfaces. |
|
batch/component/item
Package item provides various item-related components for batch processing, including readers, processors, and writers.
|
Package item provides various item-related components for batch processing, including readers, processors, and writers. |
|
batch/component/step/writer
Package writer provides implementations of the port.ItemWriter interface.
|
Package writer provides implementations of the port.ItemWriter interface. |
|
batch/component/tasklet/generic
Package generic provides Fx modules for generic tasklet components.
|
Package generic provides Fx modules for generic tasklet components. |
|
batch/component/tasklet/migration
Package migration provides the Fx module for the MigrationTasklet component.
|
Package migration provides the Fx module for the MigrationTasklet component. |
|
batch/core/application/port
Package port defines the core interfaces (ports) for the batch application.
|
Package port defines the core interfaces (ports) for the batch application. |
|
batch/core/config
Package config provides core configuration structures and utilities for the batch framework.
|
Package config provides core configuration structures and utilities for the batch framework. |
|
batch/core/config/jsl
Package jsl defines the models for the Job Specification Language (JSL) in the Surfin Batch Framework.
|
Package jsl defines the models for the Job Specification Language (JSL) in the Surfin Batch Framework. |
|
batch/core/config/support
Package support provides supporting structures and factories for the batch framework, including the central JobFactory for constructing batch components and jobs.
|
Package support provides supporting structures and factories for the batch framework, including the central JobFactory for constructing batch components and jobs. |
|
batch/core/support/expression
Package expression provides utilities for resolving dynamic expressions within the batch framework.
|
Package expression provides utilities for resolving dynamic expressions within the batch framework. |
|
batch/core/tx
Package tx provides an abstraction for transaction management in the Surfin Batch Framework.
|
Package tx provides an abstraction for transaction management in the Surfin Batch Framework. |
|
batch/engine/step/tasklet
Package tasklet provides the TaskletStep implementation for the batch engine.
|
Package tasklet provides the TaskletStep implementation for the batch engine. |
|
batch/infrastructure/repository/inmemory
Package inmemory provides an in-memory implementation of the JobRepository interface.
|
Package inmemory provides an in-memory implementation of the JobRepository interface. |
|
batch/support/util/exception
Package exception provides custom error types and error handling utilities for the Surfin Batch Framework.
|
Package exception provides custom error types and error handling utilities for the Surfin Batch Framework. |
|
batch/support/util/logger
Package logger provides a simple logging utility for the Surfin Batch Framework.
|
Package logger provides a simple logging utility for the Surfin Batch Framework. |
|
batch/support/util/serialization
Package serialization provides utilities for serializing and deserializing various data structures used in the batch framework, such as JobParameters and ExecutionContext.
|
Package serialization provides utilities for serializing and deserializing various data structures used in the batch framework, such as JobParameters and ExecutionContext. |
|
batch/test
Package test provides mock implementations and helper functions for testing various components of the batch framework, particularly database interactions.
|
Package test provides mock implementations and helper functions for testing various components of the batch framework, particularly database interactions. |
Click to show internal directories.
Click to hide internal directories.