Saltbox Docker Controller (SDC)
A dependency-aware Docker container orchestrator for Saltbox, written in Go.
Overview
SDC manages Docker container startup and shutdown based on dependency labels. It ensures containers start in the correct order, waits for health checks, and handles graceful shutdown in reverse dependency order.
Features:
- Dependency-aware orchestration using Docker labels
- Topological sort for optimal startup/shutdown order
- Parallel execution of independent containers
- Health check polling with configurable timeouts
- Job-based API for async operations
- Block/unblock operations for maintenance windows
- REST API server mode
- Helper mode for Docker daemon lifecycle integration
- Graceful shutdown handling
- Comprehensive test suite (56 tests)
Building
Build using the Makefile:
make build
The binary will be created at build/sdc.
For all available targets:
make help
Testing
Run all tests:
make test
Run tests with coverage:
make test-coverage
Usage
Server Mode
Start the REST API server:
./build/sdc server --host 127.0.0.1 --port 3377
Or use the Makefile:
make run-server
Helper Mode
Run the helper daemon for automatic lifecycle management:
./build/sdc helper --controller-url http://127.0.0.1:3377
Or use the Makefile:
make run-helper
./build/sdc --version
Architecture
Package Structure
sdc/
├── cmd/controller/ # Main entry point (server/helper commands)
├── internal/
│ ├── api/ # HTTP handlers, middleware, and router
│ ├── client/ # HTTP client for helper mode
│ ├── docker/ # Docker client wrapper and label parsing
│ ├── graph/ # Dependency graph and topological sort
│ ├── orchestrator/ # Container orchestration engine
│ ├── jobs/ # Job manager with worker pool
│ └── config/ # Configuration management
└── pkg/logger/ # Structured logging (Zap)
How It Works
- Label-based dependencies: Containers declare dependencies via
sdc.requires labels
- Graph building: SDC builds a dependency graph from all running containers
- Topological sort: Determines optimal startup/shutdown order
- Batch execution: Independent containers in each batch start/stop in parallel
- Health checking: Polls container health status before proceeding to dependents
- Job tracking: All operations are tracked as jobs with UUID and status
Dependencies
github.com/moby/moby/client v0.1.0-beta.3 - Docker client SDK
github.com/moby/moby/api v1.52.0-beta.4 - Docker API types
github.com/go-chi/chi/v5 - HTTP router
go.uber.org/zap - Structured logging
github.com/spf13/cobra - CLI framework
github.com/google/uuid - UUID generation
github.com/stretchr/testify - Testing utilities
Docker Labels
SDC uses Docker labels to define container dependencies:
labels:
sdc.requires: "postgres,redis" # Comma-separated list of container names
sdc.startup_delay: "5" # Seconds to wait after this container starts
Example docker-compose.yml:
services:
postgres:
image: postgres:15
labels:
sdc.startup_delay: "10"
redis:
image: redis:7
labels:
sdc.startup_delay: "5"
app:
image: myapp:latest
labels:
sdc.requires: "postgres,redis"
sdc.startup_delay: "2"
SDC will ensure postgres and redis start first (in parallel), wait for health checks, apply startup delays, then start app.
API Endpoints
Container Operations
POST /start - Start containers in dependency order
- Request:
{"timeout": 600, "ignore": ["container1"]}
- Response:
{"id": "uuid", "status": "pending"}
- Returns HTTP 503 if operations are blocked
POST /stop - Stop containers in reverse dependency order
- Request:
{"timeout": 300, "ignore": ["container2"]}
- Response:
{"id": "uuid", "status": "pending"}
- Returns HTTP 503 if operations are blocked
Block/Unblock Operations
POST /block/{duration} - Block start/stop operations temporarily
duration parameter in minutes (default: 10)
- Response:
{"message": "Operations are now blocked for N minutes"}
- Auto-unblocks after the specified duration
POST /unblock - Manually unblock operations
- Response:
{"message": "Operations are now unblocked"}
Job Status
GET /job_status/{job_id} - Get job details and status
- Response: Full job object with status, results, and timing information
- Returns
{"status": "not_found"} with HTTP 404 if job doesn't exist
Health
GET /ping - Health check endpoint
Helper Mode Details
The helper mode is designed to run as a systemd service for automatic container lifecycle management:
Lifecycle:
- Wait for controller server to become ready (60s timeout)
- Apply configured startup delay
- Submit start job for all managed containers
- Wait for job completion
- Run until receiving SIGTERM/SIGINT
- Submit stop job for all containers
- Wait for stop completion and exit gracefully
Blocked Operations Handling:
When start/stop operations are blocked (HTTP 503 response), the helper will:
- Log an INFO message: "Container start/stop operation is currently blocked, skipping"
- Continue running without failing
- This allows the helper to gracefully handle maintenance windows
Options:
./build/sdc helper \
--controller-url http://localhost:3377 \ # Controller API URL
--startup-delay 10s \ # Delay before starting containers
--timeout 600 \ # Job timeout in seconds
--poll-interval 5s # Status polling interval
License
GNU General Public License v3.0