dagu

README

Dagu — a pragmatic Directed Acyclic Graph Utility

Website | Quick Start | Discussions

What is Dagu?

Dagu is a DAG workflow orchestration engine that executes DAGs (Directed Acyclic Graphs) without requiring you to write Python code. The name combines "DAG" with "u" (utility), reflecting its purpose as a comprehensive utility for DAG management.

With Dagu, you can define intricate pipelines using a simple, declarative YAML format – no Python boilerplate required. This code-free approach makes your workflows incredibly easy to create, read, and maintain, empowering everyone on your team to build and understand them.

It is equipped with many features to meet the highly detailed requirements of enterprise environments.

How it Works

Dagu executes your workflows, which are defined as a series of steps in a YAML file. These steps form a Directed Acyclic Graph (DAG), ensuring a clear and predictable flow of execution.

For example, a simple sequential DAG:

schedule: "0 0 * * *" # Runs at 00:00 everyday

steps:
  - echo "Hello, dagu!"
  - echo "This is a second step"

With parallel steps:

schedule: "0 0 * * *" # Runs at 00:00 everyday

steps:
  - echo "Step 1"
  - 
    - echo "Step 2a - runs in parallel"
    - echo "Step 2b - runs in parallel"
  - echo "Step 3 - waits for parallel steps"

Or with explicit dependencies:

schedule: "0 0 * * *" # Runs at 00:00 everyday

steps:
  - name: step 1
    command: echo "Hello, dagu!"
  - name: step 2
    command: echo "This is a second step"
    depends: step 1

Highlights

• ✨ Simplicity First: Define workflows in plain YAML. If you can write a list, you can build a DAG. No Python code required.
• 🚀 Zero-Dependency Installation: Get started instantly with a single binary. No database or message queue required.
• 🌐 Distributed & Scalable: Run in High-Availability (HA) mode and distribute tasks to a fleet of workers using labels.
• 🎮 Intuitive UI & CLI: Visualize, monitor, and manage your workflows through a clean web interface or a powerful command-line tool.
• 📦 Built-in Power: Natively supports running shell commands, containers, and remote commands over SSH without extra plugins.

CLI Preview

Web UI Preview

Table of Contents

• Quick Start
  • Quick Start
  • Docker-Compose
• Documentation
  • Documentation
  • Environment Variables
  • Architecture
  • Roadmap
• Development & Contributing
  • Development
  • Discussion
  • Contributing
  • Acknowledgements
• License

Quick Start

1. Install dagu

npm:

# Install via npm
npm install -g dagu

Homebrew:

brew install dagu-org/brew/dagu

# Upgrade to latest version
brew upgrade dagu-org/brew/dagu

macOS/Linux:

# Install via script
curl -L https://raw.githubusercontent.com/dagu-org/dagu/main/scripts/installer.sh | bash

Docker:

docker run --rm \
  -v ~/.dagu:/var/lib/dagu \
  -p 8080:8080 \
  ghcr.io/dagu-org/dagu:latest \
  dagu start-all

Note: see documentation for other methods.

2. Create your first workflow

Note: When you first start Dagu with an empty DAGs directory, it automatically creates example workflows to help you get started. To skip this, set DAGU_SKIP_EXAMPLES=true.

cat > ./hello.yaml << 'EOF'
steps:
  - echo "Hello from Dagu!"
  - echo "Running step 2"
EOF

3. Run the workflow

dagu start hello.yaml

4. Check the status and view logs

dagu status hello

5. Explore the Web UI

dagu start-all

Visit http://localhost:8080

Docker-Compose

Dagu supports running multiple services in high-availability (HA) mode.

1. Create a compose.yml file in your project directory.

services:
  dagu-1:
    image: ghcr.io/dagu-org/dagu:latest
    volumes:
      - ./dagu:/var/lib/dagu
    environment:
      - DAGU_COORDINATOR_PORT=50055
    ports:
      - 8080:8080
      - 50055:50055
    command: ["dagu", "start-all"]

  dagu-2:
    image: ghcr.io/dagu-org/dagu:latest
    volumes:
      - ./dagu:/var/lib/dagu
    environment:
      - DAGU_COORDINATOR_PORT=50056 # Use a different port for the second instance
    ports:
      - 8081:8080
      - 50056:50056
    command: ["dagu", "start-all"]

volumes:
  dagu:
    driver: local

Notes:

• If the primary scheduler instance fails, the standby instance will automatically take over. The web servers are stateless and can be scaled horizontally using shared storage. For more details, see the High Availability documentation.
• Port 50055 is used for gRPC communication between the coordinator and its workers. For more information, refer to the Distributed Execution documentation.
• Dagu uses a shared volume for service registry, so you must set a unique DAGU_COORDINATOR_PORT for each service instance if they are running on the same host.
• If you are not running separate workers and instead execute DAGs locally on the scheduler instances, you can omit the DAGU_COORDINATOR_PORT environment variable.

2. Start Docker Compose

You can now start Dagu by typing:

docker compose up -d

To stop the containers, run:

docker compose stop

Documentation

Full documentation is available at docs.dagu.cloud.

Helpful Links:

• Feature by Examples - Explore useful features with examples
• Remote Execution via SSH - Run commands on remote machines using SSH
• Distributed Execution - How to run workflows across multiple machines
• Scheduling - Learn about flexible scheduling options (start, stop, restart) with cron syntax
• Authentication - Configure authentication for the Web UI
• Configuration - Detailed configuration options for customizing Dagu

Changelog

For a detailed list of changes, bug fixes, and new features, please refer to the Changelog.

Environment Variables

Note: Configuration precedence: Command-line flags > Environment variables > Configuration file

Frontend Server Configuration

Environment Variable	Default	Description
DAGU_HOST	127.0.0.1	Web UI server host
DAGU_PORT	8080	Web UI server port
DAGU_BASE_PATH	-	Base path for reverse proxy setup
DAGU_API_BASE_URL	/api/v2	API endpoint base path
DAGU_TZ	-	Server timezone (e.g., Asia/Tokyo)
DAGU_DEBUG	false	Enable debug mode
DAGU_LOG_FORMAT	text	Log format (text or json)
DAGU_HEADLESS	false	Run without Web UI
DAGU_LATEST_STATUS_TODAY	false	Show only today's latest status
DAGU_WORK_DIR	-	Default working directory for DAGs
DAGU_DEFAULT_SHELL	-	Default shell for command execution
DAGU_CERT_FILE	-	TLS certificate file path
DAGU_KEY_FILE	-	TLS key file path

Path Configuration

Environment Variable	Default	Description
DAGU_HOME	-	Base directory that overrides all path configurations
DAGU_DAGS_DIR	~/.config/dagu/dags	Directory for DAG definitions
DAGU_LOG_DIR	~/.local/share/dagu/logs	Directory for log files
DAGU_DATA_DIR	~/.local/share/dagu/data	Directory for application data
DAGU_SUSPEND_FLAGS_DIR	~/.local/share/dagu/suspend	Directory for suspend flags
DAGU_ADMIN_LOG_DIR	~/.local/share/dagu/logs/admin	Directory for admin logs
DAGU_BASE_CONFIG	~/.config/dagu/base.yaml	Path to base configuration file
DAGU_EXECUTABLE	-	Path to dagu executable
DAGU_DAG_RUNS_DIR	{dataDir}/dag-runs	Directory for DAG run data
DAGU_PROC_DIR	{dataDir}/proc	Directory for process data
DAGU_QUEUE_DIR	{dataDir}/queue	Directory for queue data
DAGU_SERVICE_REGISTRY_DIR	{dataDir}/service-registry	Directory for service registry

Authentication

Environment Variable	Default	Description
DAGU_AUTH_BASIC_USERNAME	-	Basic auth username
DAGU_AUTH_BASIC_PASSWORD	-	Basic auth password
DAGU_AUTH_OIDC_CLIENT_ID	-	OIDC client ID
DAGU_AUTH_OIDC_CLIENT_SECRET	-	OIDC client secret
DAGU_AUTH_OIDC_CLIENT_URL	-	OIDC client URL
DAGU_AUTH_OIDC_ISSUER	-	OIDC issuer URL
DAGU_AUTH_OIDC_SCOPES	-	OIDC scopes (comma-separated)
DAGU_AUTH_OIDC_WHITELIST	-	OIDC email whitelist (comma-separated)

UI Configuration

Environment Variable	Default	Description
DAGU_UI_NAVBAR_COLOR	#1976d2	UI header color (hex or name)
DAGU_UI_NAVBAR_TITLE	Dagu	UI header title
DAGU_UI_LOG_ENCODING_CHARSET	utf-8	Log file encoding
DAGU_UI_MAX_DASHBOARD_PAGE_LIMIT	100	Maximum items on dashboard
DAGU_UI_DAGS_SORT_FIELD	name	Default DAGs sort field
DAGU_UI_DAGS_SORT_ORDER	asc	Default DAGs sort order

Scheduler Configuration

Environment Variable	Default	Description
DAGU_SCHEDULER_PORT	8090	Health check server port
DAGU_SCHEDULER_LOCK_STALE_THRESHOLD	30s	Scheduler lock stale threshold
DAGU_SCHEDULER_LOCK_RETRY_INTERVAL	5s	Lock retry interval
DAGU_SCHEDULER_ZOMBIE_DETECTION_INTERVAL	45s	Zombie DAG detection interval (0 to disable)
DAGU_QUEUE_ENABLED	true	Enable queue system

Worker Configuration

This configuration is used for worker instances that execute DAGs. See the Distributed Execution documentation for more details.

Environment Variable	Default	Description
DAGU_COORDINATOR_HOST	127.0.0.1	Coordinator gRPC server host
DAGU_COORDINATOR_PORT	50055	Coordinator gRPC server port
DAGU_WORKER_ID	-	Worker instance ID
DAGU_WORKER_MAX_ACTIVE_RUNS	100	Maximum concurrent runs per worker
DAGU_WORKER_LABELS	-	Worker labels (format: key1=value1,key2=value2, e.g., gpu=true,memory=64G)
DAGU_SCHEDULER_PORT	8090	Scheduler health check server port
DAGU_SCHEDULER_LOCK_STALE_THRESHOLD	30s	Time after which scheduler lock is considered stale
DAGU_SCHEDULER_LOCK_RETRY_INTERVAL	5s	Interval between lock acquisition attempts

Peer Configuration

This configuration is used for communication between coordinator services and other services (e.g., scheduler, worker, web UI). See the Distributed Execution documentation for more details.

Environment Variable	Default	Description
DAGU_PEER_CERT_FILE	-	Peer TLS certificate file
DAGU_PEER_KEY_FILE	-	Peer TLS key file
DAGU_PEER_CLIENT_CA_FILE	-	Peer CA certificate file for client verification
DAGU_PEER_SKIP_TLS_VERIFY	false	Skip TLS certificate verification for peer connections
DAGU_PEER_INSECURE	true	Use insecure connection (h2c) instead of TLS

Development

Building from Source

Prerequisites

• Go 1.24+
• Node.js
• pnpm

1. Clone the repository and build server

git clone https://github.com/dagu-org/dagu.git && cd dagu
make

This will start the dagu server at http://localhost:8080.

2. Run the frontend development server

cd ui
pnpm install
pnpm dev

Navigate to http://localhost:8081 to view the frontend.

Roadmap

Legend:

• Status: ✅ Done / 🏗️ In Progress / 📋 Planned / 💭 Designing / ⛔ Blocked / 🏢 Enterprise
• Priority: P0 = Must have / P1 = Should have / P2 = Could have

Category	Capability	Description	Status	Priority	Link
Core Execution & Lifecycle	Local execution	Run workflows locally with CLI / Web UI / API	✅		CLI, Web UI, API
	Queue based execution	Dispatch DAG execution to workers with labels and priorities	✅		Queues
	Immediate execution	Disable queue for immediate execution	✅		CLI
	Idempotency	Prevent duplicate DAG execution with same DAG-run ID	✅		start command
	Status management	queued → running → success/partial_success/failed/cancelled	✅		Status Management
	Cancel propagation	Cancel signals to sub-DAG	✅
	Cleanup hooks	Define cleanup processing with onExit handlers	✅		Lifecycle Handlers
	Status hooks	Define hooks on success / failure / cancel	✅		Lifecycle Handlers
Definition & Templates	Declarative YAML DSL	Validation with JSON Schema, display error locations	✅		YAML Specification
	Environment variables	Environment variables at DAG and step level, support dotenv	✅		Environment Variables
	Command substitution	Use command output as value for variables or parameters	✅		Command Substitution
	Shell support	Use shell features like pipes, redirects, globbing, etc.	✅		Shell Executor
	Script support	Use scripts in Python, Bash, etc. as steps	✅		Script Execution
	Modular DAGs	Reusable DAGs with params	✅		Base Configuration, Parallel Execution
	Secrets management	Reference-only secrets via KMS/Vault/OIDC	💭	P0	#798
	Variables store	Env-scoped variables	💭	P1
	Code-based SDK	Python / Go / TS SDK to build DAG programmatically	💭	P2	#583
	Go template support	Use Go templates in DAG definitions	💭	P2	#738
Control Structures	Fan-out/Fan-in	Native parallel branches + join	✅		Parallel Execution
	Iteration (loop)	Iteration over list values	✅		Parallel Execution
	Matrix runs	Create all parameter combinations	💭	P1	#879, #990
	Conditional routes	Data/expression based routing	✅		Conditional Execution
	Sub-DAG call	Reusable sub-DAG	✅		Parallel Execution
	Worker & Dispatch	Runs DAG on different nodes with selector conditions	✅		Distributed Execution
	Human-in-the-loop	Wait for human approval / input	💭	P0	#978
	Retry policies	Retry with backoff/interval	✅		Retry Policies
	Repeat Policies	Repeat step until condition is met	✅		Repeat Policies
	Timeout management	DAG Execution Timeouts	✅		Workflow Timeout
Triggers & Scheduling	Cron expression	Schedule to start / stop / restart	✅		Scheduling
	Multiple schedules	Multiple schedules per DAG	✅		Multiple Schedules
	Timezone support	Per-DAG timezone for cron schedules	✅		Timezone Support
	Skip	Skip an execution when a previous manual run was successful	✅		Skip Redundant Runs
	Second-precision cron	Per-DAG TZ, holiday calendar, exclusion windows	💭	P2	#676
	Sunset/Sunrise triggers	Trigger DAG on sunrise/sunset events	💭	P2	#1004
	Catch up	Catch up on missed executions with safety caps	💭	P0	#695
	Overlap	Overlap policy (skip/queue/cancel)	📋	P1
	Queue Override	Override queue for specific runs	📋	P0	#1111
	Backfill	Historical range runs with safety caps	💭	P2	#695
	Zombie detection	Automatic detection for processes terminated unexpectedly	✅		Scheduling
	Trigger via Web API	Web API to start DAG executions	✅		Web API
	File watcher trigger	Trigger DAG on file changes in a directory	💭	P0	#372
	Holiday calendars	Import & reference holiday calendars	💭	P2	#676
Container Native Execution	Step-level container config	Run steps in Docker containers with granular control	✅		Docker Executor
	DAG level container config	Run all steps in a container with shared volumes and env vars	✅		Container Field
	On-the-fly image build	Builds image on-the-fly during DAG execution	📋	P0
	Authorized registry access	Access private registries with credentials	✅		Registry Auth
	Kubernetes native execution	Run steps as Kubernetes jobs/pods	💭	P0	#837
Resource Management	Resource limits	CPU/Memory/IO requests & limits per-step	💭	P0
	Rate limiting	Token bucket per key/endpoint for external APIs	💭	P1
	Distributed locks	Keyed semaphore for exclusivity	💭	P0
Data & Artifacts	Passing data between steps	Passing ephemeral data between steps in a DAG	✅		Data Flow
	JSON Schema validation	Parameter validation with JSON Schema	💭	P0	#325
	External storage	Stream large logs/artifacts to S3/GCS/Azure	💭	P2	#640, #548, #267
	Secret redaction	Auto-mask secrets in logs/events	📋	P0
	Inter DAG-run state management	Manage state and data sharing between DAG-runs	💭	P0
	Automatic log cleanup	Automatic log cleanup based on retention policies	✅		Log Retention
	Database backend support	Support for external databases (PostgreSQL, MySQL) instead of filesystem	💭	P1	#539, #267
Observability	Logging with live streaming	Structured JSON logs with live tail streaming	✅		Log Viewer
	Metrics	Prometheus metrics	✅		Metrics
	OpenTelemetry	Distributed tracing with OpenTelemetry	✅		OpenTelemetry
	DAG Visualization	DAG / Gantt charts for critical path analysis	✅		DAG Visualization
	Email notification	Email notification on success / failure with the log file attachment	✅		Email Notifications
	Health monitoring	Health check for scheduler & failover	✅		Health Check
	Nested-DAG visualization	Nested DAG visualization with drill down functionality	✅		Nested DAG Visualization
	Resource usage monitoring	CPU/Memory/IO usage per DAG/step with live graphs	💭	P0	#546
Security & Governance	Secret injection	Vault/KMS/OIDC ref-only; short-lived tokens	💭	P0	#798
	Authentication	Basic auth / OIDC support for Web UI and API	✅		Authentication
	Authorization	User management & RBAC with fine-grained permissions	🏢
	Resource quotas	CPU time and memory limit	📋	P0
	Audit trail	Immutable events for all manual actions	🏢
	Audit logging	Immutable who/what/when records (WORM)	🏢
	HA (High availability) mode	Control-plane with failover for scheduler / Web UI / Coordinator	✅		High Availability
Executor types	jq	JSON processing with jq queries	✅		JQ Executor
	ssh	Remote command execution via SSH	✅		SSH Executor
	docker	Container-based task execution	✅		Docker Executor
	http	HTTP/REST API calls with retry	✅		HTTP Executor
	mail	Send emails with template	✅		Mail Executor
	archive	Archive/unarchive operations (zip, tar, etc.)	📋	P1	#1079
	database	Direct database read/write operations	💭	P2	#789
	ftp	File transfer protocol support	💭	P2	#1079
	Custom plugin system	Custom executor types	💭	P1	#583
DevX & Testing	Local development	offline runs	✅		CLI Usage
	Dry-run	DAG level Dry-run	✅		dry command
	Debug mode	debug mode for step-by-step DAG execution	💭	P1	#1119
	Static analysis	DAG Validation tool	💭	P0	#325
	Migration helpers	Provide migration helpers from cron-only DAGs to full scheduler	💭	P1	#448
UI & Operations	Run / retry / cancel operations	Start / enqueue / retry / stop	✅		DAG Operations
	Pause / resume operations	Pause / resume DAG executions	💭	P0
	Automatic parameter forms	Auto-generate parameter forms for DAGs	✅		Web UI
	DAG definition search	Filter by tag / name	✅		DAG Search
	Execution history search	Filter by status / date-range / name	✅		History Search
	Step-level operations	Rerun, resume from step	✅		Web UI
	Run single step	Run a single step in a DAG	💭	P0	#1047
	Parameter override	Override parameters for a DAG run	✅
	Edit & retry	Edit DAG definition before retrying a run	💭	P0	#326, #1058
	Scheduled DAG management	Enable/disable schedule for a DAG	✅		Web UI
	Version control	Diff/compare/rollback DAG definitions	💭	P2	#320, #374
	UI organization	Logical DAG grouping	✅		DAG Organization
Others	Windows support	Windows support	📋	P1	#749, #1061, #859
	Snap packaging	Snap packaging	📋	P1	#821, #871

Discussion

For discussions, support, and sharing ideas, join our community on Discord.

Recent Updates

Changelog of recent updates can be found in the Changelog section of the documentation.

Contributing

We welcome contributions of all kinds! Whether you're a developer, a designer, or a user, your help is valued. Here are a few ways to get involved:

• ⭐ Star the project on GitHub.
• 🤔 Suggest new features by opening an issue.
• 👋 Join the discussion on our Discord server.
• 🧑‍💻 Contribute code: Check out our good first issue label for tasks perfect for new contributors.

For more details, see our Contribution Guide and our public Roadmap to see what's planned.

Acknowledgements

Contributors

Thanks to all the contributors who have helped make Dagu better! Your contributions, whether through code, documentation, or feedback, are invaluable to the project.

Sponsors & Supporters

Thanks for supporting Dagu’s development! Join our supporters: GitHub Sponsors

License

GNU GPLv3 - See LICENSE

---

If you find Dagu useful, please ⭐ star this repository