brokoli

command module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 7, 2026 License: GPL-3.0 Imports: 3 Imported by: 0

README

Brokoli

Brokoli

Data orchestration that deploys in 30 seconds, not 30 minutes.
Single binary. Visual editor. Built-in data quality. No infrastructure required.

Quick Start · Features · Why Brokoli · Enterprise · API


Quick Start

Option A: Binary

# Download and run — that's it
curl -L https://github.com/hc12r/brokoli/releases/latest/download/brokoli-linux-amd64 -o brokoli
chmod +x brokoli
./brokoli serve

Open http://localhost:8080. Create your admin account. Build your first pipeline.

Option B: Python SDK

pip install brokoli
from brokoli import Pipeline, task, source_api, quality_check, sink_file

with Pipeline("my_pipeline", schedule="0 6 * * *") as p:

    data = source_api("Fetch", url="https://api.example.com/data", retries=3)

    @task("Transform")
    def clean(raw):
        return [r for r in raw if r.get("status") == "active"]

    cleaned = clean(data)
    quality_check("Validate", cleaned, rules=["not_null(id)", "unique(id)"])
    cleaned >> sink_file("Save", path="/tmp/output.csv")
brokoli deploy my_pipeline.py --server http://localhost:8080
# Pipeline appears in the visual editor instantly

See the Python SDK documentation and 7 runnable demo pipelines.

From source:

cd broked-ui && npm install && npm run build && cd ..
cd broked && go build -o brokoli . && ./brokoli serve

Why Brokoli

Airflow Dagster Prefect Brokoli
Setup Docker + Redis + Celery + Postgres Docker + Dagit + Daemon Cloud signup or Docker ./brokoli serve
Binary size ~500MB with deps ~400MB Cloud-dependent 26MB
First pipeline ~30 min (write Python DAG) ~20 min (write Python assets) ~15 min (write Python flow) ~2 min (drag and drop)
Visual editor No No No Yes
Data preview No Limited No Every node, every run
Data quality External (Great Expectations) External External Built-in (10 rules)
Auto-profiling No No No Yes — null%, unique%, min/max/mean per column
Schema drift detection No No No Yes — alerts on column changes
Self-hosted Complex Complex Limited One file

Features

Visual Pipeline Editor

Drag-and-drop pipeline builder with 13 node types. No code required for common ETL patterns — but full Python support when you need it.

  • SVG canvas with zoom, pan, snap-to-port connections
  • Undo/redo, auto-layout, node duplication
  • Inline node config with connection selector
  • Dry-run preview with data at every node
  • YAML import/export for pipeline-as-code

13 Node Types

Sources Processing Outputs Flow Control
File (CSV/JSON/Excel) Transform (filter, sort, rename, aggregate) File Output If/Else Condition
REST API Python Code Database Sink
Database (Postgres/MySQL/SQLite) Join (inner/left/right/full) API Sink
Quality Check (10 rules) DB Migration
SQL Generate

Data Quality — Built In, Not Bolted On

  • 10 quality rules: not_null, unique, min, max, range, regex, row_count, type_check, freshness, no_blank
  • Auto-profiling: every node output gets profiled — row count, null%, unique%, min/max/mean, type inference, cardinality
  • Schema drift detection: automatic comparison against previous runs — alerts on column added/removed, type changes, null rate spikes
  • Quality trends: profile data stored as time series for historical analysis
  • Block or warn policies per rule

Smart Execution

  • Parallel DAG execution — wave-based Kahn's algorithm with configurable concurrency
  • Smart retry per node — exponential/linear/fixed backoff, node-type-aware defaults (DB sources get 3 retries, transforms get 0)
  • Resume from failure — skip succeeded nodes, retry from the point of failure
  • Cross-pipeline dependencies — depends_on field, scheduler checks before triggering
  • Condition branching — if/else nodes evaluate expressions (row_count > 0, null_pct("col") < 10) and route data accordingly
  • Cancellation — cancel running pipelines via API or UI

Monitoring & Alerts

  • Real-time WebSocket updates — live run status, node progress, log streaming
  • Slack notifications — configurable via UI, fires on run success, failure, and SLA breach
  • Schema drift alerts — critical schema changes trigger Slack notifications with column details
  • SLA deadlines — per-pipeline "must complete by HH:MM" with timezone support
  • Run calendar — GitHub-style heatmap of daily run activity
  • Data lineage — interactive cross-pipeline DAG visualization
  • Gantt timeline — node-level execution timing per run

Connections & Secrets

  • 7 connection types: Postgres, MySQL, SQLite, HTTP, SFTP, S3, Generic
  • AES-256-GCM encryption for all secrets at rest
  • Real connection testing — actually connects and authenticates, not just ping
  • Variables with typed values (string, number, JSON, secret) and ${var.key} resolution in any config field

Pipeline Versioning

  • Auto-snapshot on every save
  • Version history with timestamps
  • One-click rollback to any version

Webhook Triggers

  • Per-pipeline webhook tokens
  • Trigger runs via HTTP: POST /api/pipelines/:id/webhook?token=whk_...
  • Use with GitHub Actions, dbt, Kafka consumers, or any external event

CLI for CI/CD

# Trigger a run and wait for completion
brokoli run <pipeline-id> --server http://localhost:8080

# Run with assertions (exits non-zero on failure)
brokoli assert <pipeline-id> -a assertions.yaml

Assertion file format:

assertions:
  - name: "Has data"
    type: min_rows
    value: "1"
  - name: "ID is unique"
    type: unique
    column: id
  - name: "Email not null"
    type: no_nulls
    column: email

Python Integration

Works with any python3. For large datasets:

pip install pyarrow pandas
Dataset size Method Speed
< 10K rows JSON stdin/stdout Baseline
≥ 10K rows CSV temp files 3–5x faster
≥ 10K rows + pyarrow Arrow IPC 5–10x faster

Auto-detected. No configuration needed.

Enterprise

Brokoli Enterprise adds governance, compliance, and team features for production deployments. Same single binary — just with a license key.

Feature Community Enterprise
Visual pipeline editor ✓ ✓
13 node types + data quality ✓ ✓
Auto-profiling + schema drift ✓ ✓
Slack alerts (via UI config) ✓ ✓
SLA monitoring ✓ ✓
Webhook triggers + CLI ✓ ✓
Pipeline versioning + rollback ✓ ✓
Cross-pipeline dependencies ✓ ✓
Smart retry per node ✓ ✓
Condition branching (if/else) ✓ ✓
SSO/OIDC (Okta, Azure AD, Google) ✓
Audit logging with before/after diff ✓
Workspaces with RBAC (owner/admin/editor/viewer) ✓
API tokens per workspace ✓
Git Sync — pipeline-as-code with auto-deploy ✓
Data contracts — schema enforcement between teams ✓
Column-level lineage tracking ✓
PII detection — auto-scan for email, phone, SSN, IP, credit card ✓
OpenLineage export to DataHub/Marquez/Atlan ✓
Kubernetes executor — dispatch nodes as K8s Jobs ✓
BROKOLI_LICENSE_KEY=enterprise-yourco-sso,audit,k8s,gitsync,alerts-2027-01-01-xxx \
  ./brokoli-ee serve --port 9900

API Reference

All endpoints require authentication (except /health and /api/auth/setup).

Pipelines

GET    /api/pipelines                List pipelines (lean DTO — no nodes/edges)
POST   /api/pipelines                Create pipeline
GET    /api/pipelines/:id            Get pipeline (full detail)
PUT    /api/pipelines/:id            Update pipeline
DELETE /api/pipelines/:id            Delete pipeline
POST   /api/pipelines/:id/run        Trigger run
POST   /api/pipelines/:id/dry-run    Preview with sample rows
POST   /api/pipelines/:id/clone      Duplicate pipeline
POST   /api/pipelines/:id/backfill   Backfill date range
GET    /api/pipelines/:id/export     Export as YAML
GET    /api/pipelines/:id/versions   Version history
POST   /api/pipelines/:id/rollback   Restore version
GET    /api/pipelines/:id/deps       Dependency status
GET    /api/pipelines/:id/impact     Downstream impact analysis
POST   /api/pipelines/:id/webhook    Webhook trigger (token auth)
POST   /api/pipelines/import         Import YAML/JSON
POST   /api/pipelines/bulk           Bulk enable/disable/delete

Runs

GET    /api/pipelines/:id/runs            List runs
GET    /api/runs/:id                      Run detail (with error)
POST   /api/runs/:id/resume               Resume from failure
POST   /api/runs/:id/cancel               Cancel running pipeline
GET    /api/runs/:id/logs                  Run logs
GET    /api/runs/:id/logs/export           Download logs
GET    /api/runs/:id/nodes/:nid/preview    Node data preview
GET    /api/runs/:id/nodes/:nid/profile    Node profiling data + drift alerts
GET    /api/runs/calendar                  Daily run heatmap

Connections & Variables

GET    /api/connections                    List connections
POST   /api/connections                    Create connection
POST   /api/connections/:id/test           Test connection
GET    /api/variables                      List variables
POST   /api/variables                      Create/update variable

Settings & Notifications

GET    /api/settings/notifications         Slack config (masked)
PUT    /api/settings/notifications         Save Slack config
POST   /api/settings/notifications/test    Send test Slack message

System

GET    /health                             Health check (no auth)
GET    /metrics                            Prometheus metrics (no auth)
GET    /api/system/info                    System info
GET    /api/scheduler/status               Active schedules
GET    /api/lineage                        Cross-pipeline lineage graph

Architecture

26MB single binary
├── Go backend (chi router, gorilla/websocket, SQLite/PostgreSQL)
├── Svelte 5 frontend (embedded via go:embed)
├── Pipeline engine (parallel DAG execution, retry, profiling)
└── Extension system (enterprise plugins via interface injection)
broked/
  cmd/           CLI — serve, run, assert, generate-key
  engine/        Execution, transforms, profiling, drift, conditions, retry, deps, scheduler, SLA
  api/           HTTP handlers, auth, WebSocket, middleware, rate limiting
  store/         SQLite + PostgreSQL with migrations
  crypto/        AES-256-GCM encryption
  quality/       10 data quality rules
  extensions/    Enterprise plugin interfaces
  models/        Pipeline, Run, Node, Connection, Variable, Workspace
  ui/            Embedded Svelte frontend

broked-ui/
  src/pages/     Dashboard, Pipelines, Editor, Runs, Calendar, Lineage,
                 Connections, Variables, Workspaces, Audit Log, Git Sync, Settings
  src/components/ Canvas, NodeCard, Sidebar, ConfigPanel, CodeEditor, Pagination, etc.

Tests

go test ./broked/... -v
# 219+ Go tests + 52 Python SDK tests across engine, crypto, quality, store, validation
# Covers: profiling, drift detection, conditions, assertions,
#         retry logic, dependencies, webhooks, contracts, PII, lineage

Configuration

Flag Default Description
--port / -p 8080 HTTP server port
--db ./broked.db SQLite path (or postgres://... URI)
--api-key — Enable API key authentication
Env Variable Description
BROKOLI_APP_URL Base URL for Slack deep links (default: http://localhost:8080)
BROKOLI_JWT_SECRET Persistent JWT signing secret
BROKED_SECRET_* Resolved via ${secret.name} in node configs

License

Apache 2.0 — see LICENSE.

Enterprise features are available under a separate commercial license. Contact us for pricing.

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
Package extensions defines interfaces for enterprise feature plugins.
Package extensions defines interfaces for enterprise feature plugins.
pkg

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL