conveyor

README

Conveyor

   

A distributed, push-based task queue for Go.

Persistent tasks with at-least-once execution, retries with backoff, scheduling, and priorities — backed by Postgres or an in-memory broker, with no Redis and no polling.

Quickstart · Writing a worker · SDKs · Documentation

Features

• Push-based dispatch — the server streams work to connected workers the instant it exists, with credit-based flow control. No polling.
• At-least-once with crash safety — tasks are persisted before dispatch and survive server and worker crashes; a dead worker's task is redelivered.
• Deploys are free — a worker shutting down hands its in-flight tasks back with no retry penalty and no backoff; they resume immediately elsewhere, so rolling out a new build never burns a task's retry budget.
• Retries with exponential backoff, delayed and scheduled tasks, per-task timeouts/deadlines, per-task priorities and weighted queues.
• Unique tasks, dead-letter/archive, retention, per-queue pause/resume, and a per-task-type circuit breaker.
• Expiring tasks — a pre-dispatch TTL (ExpiresIn/ExpiresAt): a task not dispatched in time is archived instead of run, for work that goes stale.
• Group aggregation — coalesce many tasks into one batch and process them in a single handler call (debounce/digest, or bulk processing); fires on size, delay, or grace period.
• Rate limiting — cap a queue's dispatch rate (token bucket: rate + burst) to protect a downstream; a global default plus per-queue overrides, tunable live from the CLI, dashboard, or API.
• SDK middleware — wrap both sides: decorate enqueues (WithEnqueueMiddleware) and handlers (Mux.Use, Mux.UseBatch) for logging, metrics, or policy, without touching task code.
• End-to-end encryption — seal task payloads in the SDK/CLI (WithEncryption) so the server stores ciphertext only and holds no keys; built-in AES-256-GCM or bring your own KMS/HSM codec.
• Cron — server-persisted schedules that survive restarts and failover, pausable at runtime.
• Built-in clustering / HA — multi-node by default; a lost node's work re-activates elsewhere with zero task loss.
• Four ways to run it — standalone, cluster, Kubernetes, or embedded in a Go process.
• Secure by default — bearer-token auth that fails closed: outside --dev the server refuses to start unauthenticated unless you opt in explicitly.
• Built-in operations dashboard — an embedded web console to inspect and operate queues, tasks, cron, and connected workers; host it anywhere.
• Prometheus metrics and OpenTelemetry traces out of the box.

Quickstart

Terminal 1 — start a development server (in-memory broker, auth off):

go run ./cmd/conveyord --dev

Terminal 2 — run a worker:

go run ./examples/standalone/worker

Terminal 3 — enqueue ten welcome emails:

go run ./examples/standalone/client

The worker prints one line per processed task. The whole flow is scripted in hack/quickstart.sh (make quickstart), which CI runs on every change under a 60-second budget.

Try it with Docker

Prebuilt multi-arch images are published to the GitHub Container Registry on every push to main (:edge) and every release (:vX.Y.Z, :latest).

Run a throwaway server (in-memory broker, auth off) in one line:

docker run --rm -p 8080:8080 ghcr.io/conveyorq/conveyor:edge --dev

Or bring up a realistic stack (server + Postgres) with Compose:

docker compose -f deploy/compose/quickstart.yaml up

Either way the API is on http://localhost:8080 (with /healthz and /readyz); the Compose stack also serves metrics on http://localhost:9464/metrics. Both run the API without authentication for local evaluation; a real deployment sets api.auth_tokens (see the operations guide).

Writing a worker

w, _ := conveyor.NewWorker("http://localhost:8080",
    conveyor.WithQueues(map[string]int{"critical": 6, "default": 3}),
    conveyor.WithConcurrency(20),
)

mux := conveyor.NewMux()
mux.HandleFunc("email:welcome", func(ctx context.Context, t *conveyor.Task) error {
    var p WelcomeEmail
    if err := t.Bind(&p); err != nil {
        return conveyor.SkipRetry(err) // a payload that cannot decode never will
    }

    return sendEmail(ctx, p)
})

_ = w.Run(ctx, mux) // blocks; reconnects with jitter; drains on ctx cancel

Handlers must be idempotent and should honor ctx.Done(). Panics are recovered and reported as retryable failures — a panicking handler never kills the worker.

Enqueueing work

client, _ := conveyor.NewClient("http://localhost:8080")

info, _ := client.Enqueue(ctx,
    conveyor.NewTask("email:welcome", conveyor.JSON(WelcomeEmail{UserID: 42})),
    conveyor.Queue("critical"),
    conveyor.ProcessIn(5*time.Minute),
    conveyor.MaxRetry(10),
)

Or from the command line:

go run ./cmd/conveyor enqueue email:welcome --queue critical --json '{"user_id":42}' --in 5m

SDKs

One wire protocol, three SDKs — a task enqueued from any of them runs on a worker written in any other.

• Go — sdks/go, import conveyor "github.com/conveyorq/conveyor/sdks/go". The reference implementation (the worker and enqueue examples above).
• TypeScript — sdks/typescript, Node 20+.
• Python — sdks/python, Python 3.9+, with both async and synchronous APIs.

The TypeScript and Python SDKs match the Go SDK feature-for-feature: a producer client, a worker runtime (push-based dispatch, heartbeats, graceful drain), JSON/binary codecs, and AES-256-GCM end-to-end encryption that is byte-compatible across all three. The SDKs are not yet published to npm or PyPI — install from the in-repo source; each SDK's README has the exact commands. The wire contract that any new SDK implements is specified in docs/protocol.md.

Embedded mode

Run the whole system — broker, server, and dispatch — inside your own Go process, with no separate conveyord to deploy and no network hop. Your handler and enqueue code is identical to a remote deployment.

system, _ := embedded.Start(ctx, embedded.Config{Broker: embedded.Memory()})
defer system.Stop(ctx)

client := system.Client()
worker := system.Worker(conveyor.WithQueues(map[string]int{"default": 1}), conveyor.WithConcurrency(8))

Use it for:

• Tests — a full-fidelity node in a plain go test, with no Docker or external infrastructure.
• Single-process apps — a CLI, an edge/desktop binary, or a small service that wants durable background jobs without operating a separate server.
• Local development and demos — producer, worker, and server in one process (see examples/embedded).
• A zero-friction start — adopt Conveyor now; graduating to a cluster later is just swapping embedded.Start for conveyor.NewClient/NewWorker with a URL, leaving handler and enqueue code unchanged.

Durability vs. availability — read this before production. Embedded is a single process, which has two distinct consequences people often conflate:

• Durability is your choice of broker. embedded.Memory() keeps nothing across a restart — right for tests and disposable work. embedded.Postgres(dsn) gives the same durability as a remote deployment: queued and in-flight work survives a restart.
• It is not highly available. One process is a single point of failure — while it is down, nothing is dispatched. Embedded always runs as a cluster of one on a private loopback port and cannot join or form a multi-node cluster.

So embedded can be durable (with Postgres) but never HA. For high availability — no single point of failure, automatic failover — run the real deployment: multiple conveyord nodes clustered over a shared Postgres broker, where a lost node's work re-activates elsewhere with no task loss. See the operations guide.

How it works

Conveyor has three moving parts: your client enqueues tasks, the server (conveyord) owns them, and your workers process them. A durable broker — Postgres in production, in-memory for dev — is the source of truth. Tasks are persisted before they're dispatched, so they survive any crash. The architecture diagram at the top of this README shows the whole flow.

Push, not poll. The server pushes tasks to workers the instant work exists and a worker has free capacity — there's no poll interval to tune and no Redis. Each worker opens one persistent connection, tells the server which queues it serves and how much it can handle, and receives work over that stream. When a worker is saturated it simply stops accepting more, and the extra work waits safely in the broker.

At-least-once execution. A task is delivered until a worker acknowledges it. If a worker dies mid-task, the task is redelivered — so handlers must be idempotent. Return conveyor.SkipRetry(err) to dead-letter immediately; panics are recovered and treated as retryable failures.

What the server gives you: named queues with weights, bounded worker concurrency, retries with exponential backoff, per-task priorities, delayed and scheduled tasks, cron, unique tasks, retention/archival, and a read-only admin/inspection API — all enforced server-side. Your code only writes handlers and enqueues tasks.

The server coordinates all of this across a cluster of conveyord nodes: queues, scheduling, and lease recovery rebalance automatically when a node is lost, and no task is dropped. Scale by adding nodes; the broker is the only stateful dependency.

Dashboard

conveyord embeds a web operations console, served at the API root and enabled by default in every mode, production included — open the server's API URL in a browser. In production it sits behind the same bearer-token auth as the API: you enter a token in the UI, which is kept client-side and sent on each call. conveyord --dev is just the local convenience — auth off, zero config.

It is a full read and write console: inspect queues, drill into tasks by state with a detail view, manage cron, and see the worker sessions connected to each node — and act: run/cancel/delete tasks, pause/resume queues, edit cron. Auto-refresh keeps it live, and a configurable link deep-links to Grafana for time-series metrics.

The UI is just an API client, so you can run the embedded copy, serve the same bundle from your own host/CDN, or front both behind one ingress. Set api.dashboard: false to disable the embedded copy, api.cors_origins to allow a different-origin UI, and api.grafana_url for the metrics link. See the operations guide.

Documentation

• Architecture — how the system works inside: the actor runtime, queue grains, broker, dispatch, and clustering, for contributors.
• Operations guide — deployment modes, configuration, scaling, broker sizing, security, observability, and upgrades.
• Group aggregation — how to enqueue grouped tasks, write batch handlers, and tune the firing policy.
• Rate limiting — cap per-queue dispatch rate with a global default and live per-queue overrides.
• End-to-end encryption — seal task payloads in the SDK/CLI so the server stores ciphertext only and holds no keys.
• Expiring tasks — a pre-dispatch TTL, and how it differs from a deadline and from retention.
• Wire protocol — the normative protocol spec for SDK authors building a Conveyor client or worker in another language.
• Go SDK — the reference client and worker for Go services.
• TypeScript SDK — enqueue and process tasks from Node.
• Python SDK — async and sync clients and workers, with a "Conveyor for Celery/RQ users" intro.
• Migrating from asynq — side-by-side API mapping.
• Migrating from River — side-by-side API mapping, and the one trade-off to decide first (transactional enqueue).
• Benchmark harness — reproducible throughput/latency harness (make benchmark) and its honesty notes.
• Deployment artifacts live under deploy/: Docker, Helm, systemd, Compose, and Grafana.
• Contributing — build, test, conventions, and how to submit changes.
• Changelog — release history.

Development

make help        # all targets
make test        # race-enabled tests (Postgres tests need Docker)
make lint        # golangci-lint via the pinned tools image
make quickstart  # the scripted README quickstart
make chaos       # 3-node kill test, repeated for the zero-loss gate (CHAOS_COUNT=20)
make e2e         # kind-based end-to-end deployment test (KEEP=1 keeps the cluster)
make e2e-demo    # live playground: cluster + continuous load + dashboard
make dashboard   # rebuild the embedded dashboard bundle (needs Node)

End-to-end deployment test

make e2e runs hack/e2e-kind.sh, which stands up a throwaway kind cluster close to a production setup: a Postgres broker, three server replicas in kubernetes mode discovering each other through the Kubernetes API, the database DSN and API tokens delivered as Secrets, and metrics on their own port. It builds the image, loads it into kind, installs the Helm chart, and asserts the rollout completes, the three nodes form one cluster, and the metrics endpoint serves. It then drives load through a rolling restart — an in-cluster producer/worker enqueues and processes tasks through the API Service while the StatefulSet is rolled one pod at a time — and asserts the cluster reforms and every task completes with zero loss. It needs docker, kind, kubectl, and helm, and runs the same way locally and in CI.

The cluster is torn down automatically on exit. To watch it live instead — stand up the cluster, run a continuous producer/worker, and open the dashboard so you can see tasks flow — use the one-command playground:

make e2e-demo      # cluster + continuous load + live dashboard at http://localhost:8080/ (token: e2e-token)
make e2e-clean     # tear the cluster down when finished

It runs the same health checks first, then goes live and blocks until Ctrl-C; turn on Auto-refresh in the UI to watch the queues, tasks, and workers update. The pieces are also available on their own: KEEP=1 make e2e (the assert-and-exit test, kept), make e2e-dashboard (port-forward + open an existing cluster's dashboard), and make e2e-clean (remove a cluster, including one left by an interrupted run).

Dashboard development

The dashboard (web/dashboard/) is a React + TypeScript app built with Vite. Its built bundle (dist/) is not committed — it's built in CI and baked into the Docker image. go build/go test don't need Node (the dashboard tests skip when the bundle is absent, and the binary serves an empty dashboard until built). Build it locally with:

make dashboard       # build web/dashboard/dist (embedded by conveyord)
make dashboard-test  # run the frontend unit tests (Vitest)
make dashboard-gen   # regenerate the TypeScript API client from the protos

For a fast edit loop, run a dev server against a local conveyord --dev:

go run ./cmd/conveyord --dev                       # API + dashboard on :8080
cd web/dashboard && npm install && npm run dev      # hot-reloading UI on :5173

Open the Vite dev server with ?api=http://localhost:8080 so it targets the running server. After changing the UI, run make dashboard to refresh the committed dist/ that ships in the binary.

License

Conveyor is fully open source under the Apache License 2.0 — the entire project, with no separate enterprise, commercial, or closed-source edition. Third-party dependency licenses are inventoried in docs/licenses.md.