workflow

package module
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

README

ubgo/workflow

Cron, delayed jobs, background queues and multi-step workflows for Go, behind one interface

One embeddable Go library for cron schedules, delayed jobs, background jobs with retries and multi-step workflows, behind one Client interface with swappable drivers.

ubgo/workflow is an open-source Go job queue, cron scheduler and durable workflow library in one. The same handlers run on an in-memory driver in tests, on SQLite, Postgres or MySQL 8 through database/sql and ubgo/sqlb, on Redis, or on the Hatchet engine, and changing the backend is a constructor change rather than a rewrite. It covers cron with time zones, delayed and one-off jobs, retries with backoff, idempotency keys, concurrency keys and rate limits, durable sleeps and wait-for-event steps, batches with callbacks, unique and debounced jobs, transactional enqueue and a transactional outbox, failover between two backends, cron overlap control and trigger-now, failure alerts and Sentry reporting, an admin API and web UI, and OpenTelemetry tracing and metrics.

Status: early (v0.x). The API is not stable yet. Every module is tagged on its own: v0.1.0 for the core, sqlstore/v0.1.0 for the sqlstore module, and the same for the others.

Contents

Why ubgo/workflow?

Every Go app that does work outside a request rebuilds the same stack: a cron runner, a job queue tied to one backend (Redis, Postgres only, or a cloud queue with hand-written polling), retries and uniqueness per app, and a workflow engine server for anything multi-step. Each choice locks the app into one backend, and code written against a workflow server cannot run without one. ubgo/workflow puts schedules, jobs and workflows behind one Client interface, so the backend becomes a deployment choice.

  • One interface, several drivers. In-memory for tests and local runs, SQLite for a single binary, Postgres or MySQL 8 for many workers over one database, Hatchet when a managed engine is wanted.
  • Honest capabilities. A driver declares what it can really do (Caps); Register (Hatchet) or Start (the other drivers) refuses a workflow that needs a capability the driver lacks, and a field the driver cannot honour returns an error wrapping ErrUnsupported that names it.
  • Proven, not claimed. Every driver runs the shared workflowtest conformance suite against the real backend; the SQL driver adds multi-client tests (each step claimed once, one cron tick per schedule across three clients, a dead worker's lease reclaimed), and the Hatchet live suite kills a worker process mid-batch.
  • Your database, your transaction. The SQL driver writes a run inside the app's own *sql.Tx (EnqueueTx), and the outbox does the same for any other driver.
  • Correct scheduling. Its own cron parser, evaluated in an explicit time zone with written DST rules, and an injected clock, so tests advance time instead of sleeping.
  • A small core. The core module imports only the standard library, enforced by deps_test.go; the SQL driver, Hatchet driver and OpenTelemetry middleware are separate modules.
  • Batteries as optional packages. Batches with Then, Catch and Finally callbacks, unique and debounced jobs, claim check for big payloads, failover between backends, retention of finished runs, and test doubles with fluent assertions.
ubgo/workflow Cron libraries Single-backend job queues Workflow engine servers
Cron in an explicit time zone ✅ ✅ ⚠️ varies ⚠️ varies
Durable background jobs with retries ✅ ❌ ✅ ✅
Multi-step workflows, durable sleep, wait for event ✅ ❌ ⚠️ varies ✅
Runs in-process with no extra server ✅ memory, SQLite ✅ ⚠️ needs Redis or Postgres in most ❌
Same code on memory, SQLite, Postgres, MySQL and a hosted engine ✅ ❌ ❌ ❌
Enqueue inside the app's own SQL transaction ✅ SQL driver, outbox ❌ ⚠️ varies ❌

If you want a single backend you already run and never expect to change, a dedicated queue for it exists; if you want the same job and workflow code to run in tests, on one SQLite file and on a cluster, that is ubgo/workflow.

Documentation

  • Learning path: fourteen steps from a first job on the memory driver to several workers on Postgres, each with its guide and a runnable example.
  • Choosing: decision tables for which driver, which method (Enqueue, EnqueueTx or the outbox; cron or Schedule; idempotency key, unique or debounce; and more), and which setup for your kind of app.
  • Feature index: every feature with its package, guide section and example.
  • Examples: 86 runnable programs, each run by go test; every Go block in the guides is copied from one of them and checked by task docs:snippets.
  • Benchmarks: enqueue throughput, end-to-end throughput and pickup latency for each driver, and how to reproduce them. On one laptop with 10 workers and a separate producer, sqlstore on Postgres picks up a new job in 7 ms median with LISTEN/NOTIFY against 415 ms when it only polls, and runs about 1,100 to 1,200 jobs/s end to end at 10 to 100 workers; the Redis driver picks up in under 1 ms and runs about 5,500 jobs/s end to end; Hatchet picks up in about 13 ms.
  • All guides: jobs, schedules, workflows, drivers, middleware, observability, batches, reliability, testing, operations, admin and extending.

Install

The core module (which holds clock, cron, memory, middleware, batch, failover and workflowtest) needs only the standard library:

go get github.com/ubgo/workflow@latest

Each separate module (sqlstore, redis, hatchet, otel, sentry, admin, cli, recorder, recorder/sqlrecorder) has its own tag and requires a tagged core, so go get github.com/ubgo/workflow/sqlstore@latest adds the module and the core it needs. Inside a clone of this repository, the committed go.work wires every module together.

Packages

Package What it is
github.com/ubgo/workflow The contract: Client, Workflow, Step (run, wait for event, sleep), Job, Schedule, retry and concurrency policies, capabilities (Caps), errors, Define for typed JSON handlers, StepInfo on the step context, the run-finished hook (RunNotifier, RunEvent), plugins (Plugin, Use: step middleware, enqueue middleware and run-finished observers bundled behind one name, each an optional interface). Retry jitter (Backoff.Jitter) and per-error delays (RetryPolicy.DelayFor, RetryAfter, RetryAfterDelay), resolved by one RetryPolicy.NextDelay; cron misfire policies (Misfire{Policy, Within, Max} with MisfireSkip, MisfireFireOnce or MisfireFireAll on Workflow and Schedule, applied by Misfire.Due; MisfireGrace 1m); Workflow.CronLocation (nil = UTC); capabilities CapRetryJitter, CapRetryDelay, CapMisfireSkip, CapMisfireFireOnce, CapMisfireFireAll and CapCronZone, checked by Requires and by RequiresSchedule/CheckScheduleCaps; ErrUnavailable with Unavailable/IsUnavailable for backend outages; the reserved label prefixes in one list (ReservedLabelPrefixes, ReservedLabel, ValidateLabels; LabelPrefixInternal ubgo.workflow/, LabelPrefixBatch workflow.batch., LabelPrefixOTel otel.), refused on Schedule.Labels. Standard library only. Optional operator and scheduling contracts, each interface behind its own capability: named queues (Workflow.Queue, Job.Queue, QueueConfig with a per-client Workers cap and a cluster-wide Limit), per-job and bounded retries with one precedence rule (Job.Retry over Step.Retry over Workflow.Retry over DefaultRetryMax, resolved by EffectiveRetriesFor; RetryPolicy.Until, MaxErrors, Discard, and DiscardOnExhaust to end a run that ran out of retries discarded instead of failed), RunQuerier (with RunQuery.Ref), RunRetrier, RunDeleter, BulkCanceller, Pauser, TickNotifier, ScheduleLister, SchedulePauser, Drainer (with DrainTimeout, ReportingDrainer and DrainReport listing the attempts a drain cut off, and ShutdownHook, a plain func(context.Context) error for any phased shutdown manager), schedule conditions (ScheduleWhen, Schedule.Windows, CronWindows), AttachScheduleHooks, RunInline, and Forward for wrappers; the retrying and discarded run states (StatusRetrying, StatusDiscarded, CapRunStates) with the whole state machine as data (Transitions), chains of jobs edited from a running step (Job.Chain, PrependToChain, AppendToChain, CapChain), worker balancing (WorkerScaler, Balancer, ByDepth, ByWait, BalanceLoop, CapWorkerScale), in-process closures (EnqueueClosure, CapClosure), and deferred dispatch (Deferred, DeferEnqueue, DeferHTTP), which holds a request's enqueues until it succeeds. Durable closures by name (NewFunc, Func, RegisterFunc, FuncJob, FuncStep, FuncQueue/FuncRetry/FuncTimeout/FuncCatch: a one-step workflow, so every driver runs it); worker recycling (Recycle, ErrRecycle, RecycleError, RecycleWatch, StopCauser, Serve); label prefix search (RunQuery.LabelPrefixes, CapLabelPrefix, CheckRunQuery). Cron overlap policy (OverlapSkip), trigger-now (ScheduleTriggerer), last fire (ScheduleInfo.Last), per-attempt error history (RunInfo.Errors), force complete and fail (RunForcer), the Reporter interface for error trackers, and the capabilities behind them (CapOverlapSkip, CapScheduleTrigger, CapScheduleHistory, CapEnqueueOnly, CapErrorHistory, CapShutdownRequeue, CapForceRun, and the three added with the queue cap, discard-on-exhaust and drain report: CapQueueLimitGlobal, CapDiscardExhausted, CapDrainReport). Also, each behind its own capability: snooze (Snooze, CapSnooze), heartbeats (Heartbeat, Step.HeartbeatTimeout), tenants (Job.Tenant, Schedule.Tenant), run timeouts (Workflow.RunTimeout, OnTimeout, the ErrTimeout family), results (ResultReader, Await, AwaitResult), ReportProgress, LifecycleNotifier, node identity and HealthChecker, RunRescheduler, StepCanceller, due-time queries, FIFO per concurrency key, payload limits and versioning (JobOf, DefineVersioned, Upcaster), AwaitChild and AwaitChildren, Memo with JournalStore, Choice, Loop, MapEach, Spawn, run data, UpstreamLimit with BucketStore, SecretGuard, PeriodKey and OncePerPeriod, ProcessChunks, schedule specs as data (Schedule JSON, cron.Calendar, NextRuns, Schedule.Seconds, Active, KeepRuns, InEnvironment, LocalCron, LabelTickAt) and Supports, with the capabilities CapSnooze, CapHeartbeat, CapTenant, CapRunTimeout, CapNodeIdentity, CapKeyFIFO, CapDueQuery, CapResult, CapCronSeconds, CapScheduleActive, CapScheduleKeep and CapTickTime.
clock The injected time source: Real, and Fake for tests (timers fire on Advance).
cron Own five-field cron parser and next-run maths in an explicit time zone. cron.Parser (Parse is the default) for drivers that accept a custom dialect. Builder writes expressions from named frequencies (cron.DailyAt("9:00").Weekdays(), EveryMinutes(15)), wrapping ErrBuilder. Six-field expressions with seconds (ParseSeconds), structured calendar specs (cron.Calendar), LoadZone with zone validation; cron/tzdata embeds the time zone database (opt-in, about 450 KB).
memory In-process driver: every capability except durability. For tests, local runs and work that may be lost on exit. WithCronParser (an opt-in custom dialect, which gives up the same-spec-on-every-driver promise) and WithJitterSource; the default misfire policy is fire-once. Implements every optional interface; WithQueues (with Workers and Limit caps), WithStopWhenIdle, DrainWithReport, and WithRunRetention (by default finished runs are kept 24 hours, at most 10,000). WithRecycle and StopCause; declares CapLabelPrefix. Overlap, trigger-now, last fire, error history (WithErrorHistory) and force; no enqueue-only mode or shutdown requeue, since its runs live in the process.
middleware Step middleware (logging, recovery, timeout, codec, error throttling), enqueue middleware (unique, unique until the run finishes, debounce, route) and the WithCodec client wrapper that seals payloads, schedule inputs and Signal payloads; claim check for big payloads (NewClaimCheck(store, threshold) with Enqueue, Step and CleanupOnFinish: payloads and step outputs above the threshold go to a pluggable BlobStore, with MemoryBlobStore built in, and a reference travels in their place). Debounce and ThrottleErrors keep their state in pluggable stores (DebounceStore via WithDebounceStore, CircuitStore via ThrottleConfig.Store; in-process MemoryDebounce/MemoryCircuit by default), so a shared store makes them fleet-wide; Logging takes WithLogLevel; Encrypt(key, oldKeys...) decrypts with retired keys for rotation. Release puts a step back without running it (ErrReleased); UniqueUntilStarted and ReleaseUniqueOnStart free a unique lock when the run starts; Pinger sends heartbeat pings from schedule hooks; WrapClient and WithCodec forward every optional client interface. Task output: CaptureOutput with OutputLogger and an OutputSink (WriterSink, FileSink replace or append, OutputSinkFunc for a mailer), result only with CaptureResult. UniqueByArgs (unique by workflow and arguments); failure alerts with dedup and rate limits (Alerts, AlertPlugin, AlertLimiter); failure reporting (Report, ReportFailures, ReportAttempts).
batch Batches of jobs with Then, Catch and Finally callbacks and progress, on any Client: run ends arrive through pluggable Sources (run-finished hook, Client.Status poller); a callback whose enqueue fails stays pending and is retried on the poller's tick, the next job end, or a retry loop Attach starts when no poller runs (WithManualCallbackRetry turns it off) under the batch's idempotency key; a client implementing batch.GroupRunner starts a batch's jobs as one native group; Store interface plus MemoryStore; a client implementing batch.GroupSpawner (Hatchet with Config.BatchCoordinator) starts the batch as one coordinator run whose children are tracked by label, which needs a LabelSource; a dispatch whose process died is resumed by Manager.ResumeDispatches (called by the poller, the retry loop and the Hatchet BatchSource) once its lease (WithDispatchLease) ends: it starts again only jobs known never to have started (a RunFinder on the client, or the per-job path's order) whose specs were kept (always on the group paths, WithResumableDispatch on the per-job path) and records the rest as cancelled. Batch.Labels are saved in State and copied onto Then, Catch and Finally jobs (reserved batch labels are refused); WithIDFunc for UUIDv7 or ULID batch IDs. Manager.Add, RetryFailed (WithRetryableJobs), Before and Progress callbacks, Prune through an optional Pruner, AwaitStep, EnqueueMany.
failover A workflow.Client over a primary and a secondary backend (failover.New(primary, secondary, opts...)): Enqueue goes to the primary and, when an injectable Classifier calls its error "unavailable" (the default never fails over on a cancelled caller ctx, core sentinels such as ErrInvalid/ErrUnsupported/ErrNotFound/ErrNotStarted, or Permanent errors), to the secondary; a job without IdempotencyKey is refused with ErrKeyRequired (matches ErrInvalid) unless AllowWithoutKey, since an ambiguous primary failure can run it on both and a key dedups within one backend only; RunRefs, RunEvents and StepInfo carry a primary:/secondary: prefix in front of the driver's own ID so Status and Cancel reach the backend holding the run; Capabilities are the intersection (CapCron the primary's alone, since Workflow.Cron and Schedule live on the primary unless WithSchedulesOnBoth, to avoid double fires); Signal and OnRunFinished go to both (CapRunFinished only when both guarantee it); WithCoolOff(n, d) skips the primary for d after n consecutive failovers on an injected clock. DefaultClassifier also fails over on errors marked workflow.Unavailable, even when they wrap context.DeadlineExceeded; StrictClassifier fails over only on those; WithObserver reports failovers, primary skips and cool-off start and end. Implements the optional run and schedule interfaces: Runs merged across backends with prefixed refs, Retry/Delete by prefix, CancelAll/Pause on both, the schedule family the primary's unless WithSchedulesOnBoth. Passes StopCause through, joining both sides; CapLabelPrefix only when both backends have it. TriggerSchedule on the primary and ForceComplete/ForceFail by ref prefix; fired runs' LabelSchedule is the prefixed ID.
workflowtest The conformance suite every driver runs (workflowtest.Run), plus app-test doubles: Fake, a recording client that runs nothing and refuses what a real driver refuses, with fluent assertions (AssertEnqueued(t, wf, ...matchers), AssertEnqueuedTimes, AssertNotEnqueued, AssertNothingEnqueued, AssertScheduled, AssertSignalled, AssertCancelled); and Recorder, which wraps a real client and records its enqueues (its standalone mode, NewRecorder(nil), is kept for compatibility; use Fake instead). FakeCronParser; the Fake enforces CheckScheduleCaps; conformance groups retryseam and misfire. AssertBatched/AssertBatchCount/AssertNothingBatched/AssertChained, FakeExcept/FakeOnly pass-through, RunStep; the Fake records Retry, Delete and Pause (AssertRetried, AssertDeleted, AssertPaused, AssertEnqueuedOn). Conformance groups func, retryingerror, labelprefix and done. The Fake records TriggerSchedule, ForceComplete and ForceFail (AssertTriggered, AssertForced); Instance.EnqueueOnly; conformance groups overlap, trigger, schedhistory, enqueueonly, errorhistory, shutdownrequeue and force.
events Same module as the core: events.Bus, publish and subscribe inside the library (a published event becomes one Enqueue per matching subscription, deduplicated by event ID; not a message broker), events.NewWebhook (an inbound handler that verifies a signature and enqueues) and HMAC signing schemes (events.HMAC, events.StandardWebhooks).
hybrid Same module as the core: hybrid.New(scheduler, jobs) fires crons on one client (usually memory) and keeps the runs on another (a durable driver), with an idempotency key per tick.
github.com/ubgo/workflow/cli Separate module (standard library only): the workflow command over the admin JSON API (runs, inspect, retry, cancel, pause, trigger, drain, metrics, health, nodes, with --json and --dry-run), and cli.Run to embed it in your own binary.
github.com/ubgo/workflow/hatchet Separate module (go 1.26): the Hatchet driver on the Hatchet Go SDK sdks/go; declares Durable, DelayedStart, WaitForEvent, DurableSleep, ConcurrencyKey, RateLimit, IdempotencyKey, Cron, Cancel, Queue, RunQuery, RetryRun, CancelAll, ScheduleList, Drain, DrainReport (PauseWorkflow and SchedulePause only with Config.API); refuses a non-zero QueueConfig.Limit and RetryPolicy.DiscardOnExhaust; implements RunNotifier best-effort (no CapRunFinished); quirks and refusals in hatchet/doc.go; implements batch.GroupRunner as bulk triggers (one RunMany per chunk of up to 1000 jobs of one workflow and about 2 MiB of input) and offers Client.BatchSource(store, interval, logger), a batch completion source that makes one run-list call per open batch, filtered by its batch label, instead of one Status call per job; implements batch.RunFinder (FindBatchRuns, by label) for resumed dispatches; with Config.BatchCoordinator implements batch.GroupSpawner: one coordinator run per batch spawns the jobs as children keyed by index (a retried coordinator gets existing children back, proven by a SIGKILL test), CancelGroup cancels by label because the engine does not cascade a parent cancel to children, over about 2 MiB of input falls back to bulk triggers. Config tunings with today's defaults: GroupChunkRuns/GroupChunkBytes (bulk chunk of at most 1000 runs and 2 MiB, refused above), BatchCoordinatorMaxInput, BatchCoordinatorRetry (a *workflow.RetryPolicy on Hatchet's factor^n formula), RegisterTimeout, VisibilityLag; Config.Logger (nil discards) gets background errors (run-finished polls, observer panics, worker stop) and is BatchSource's default logger; Config.Clock drives poll tickers, the delivered-run window and the FinishedAt of a delayed start cancelled before it fired, while engine-compared times stay wall-clock; workflows are refused at Register; refuses a non-UTC CronLocation/Schedule.Location, any Misfire policy (Hatchet never catches up), Backoff.Jitter and RetryPolicy.DelayFor, and declares none of their capabilities; errors from an unreachable engine (gRPC Unavailable, network errors, HTTP 502/503/504) match workflow.ErrUnavailable. Named queues (Config.Queues: one Hatchet worker per queue), Runs, Retry (Hatchet replay), CancelAll, Schedules, Drain; a named Schedule upserts by name; Config.API adds workflow and schedule pause. Refuses run states, Job.Chain, worker scaling and closures (no CapRunStates, CapChain, CapWorkerScale or CapClosure): a run in backoff reads queued and a discarded run failed. Config.Recycle (worker recycling: workers paused on the engine, up to Slots attempts may start past MaxJobs) and StopCause; refuses RunQuery.LabelPrefixes; runs Funcs with no driver change. TriggerSchedule, ScheduleInfo.Last read from the engine's run list (Config.ScheduleHistoryWindow) and Config.EnqueueOnly, each proven live; refuses OverlapSkip; no error history, shutdown requeue or force. Also: CapTenant, CapTickTime, CapDueQuery, CapResult, CapCronSeconds and CapKeyFIFO proven live, plus Reschedule, Nodes, Health and Config.PayloadLimit; refuses Snooze, heartbeats, run timeouts, Active, KeepRuns and queue weights, and keeps no per-attempt node history.
github.com/ubgo/workflow/otel Separate module (go 1.25): OpenTelemetry step and enqueue middleware: a span per step attempt and per Enqueue, trace context carried from Enqueue to step in otel.-prefixed job labels, step duration and outcome metrics and an enqueue counter; job labels recorded as span attributes only on opt-in (WithLabelFilter, WithAllLabels); injected TracerProvider and MeterProvider. WithParenting(ParentChild, ParentLink or ParentNewRootLink) for span links instead of parent and child; WithStepSpanName/WithEnqueueSpanName; WithMetricLabels(keys...) for opt-in, named-only label dimensions on metrics; Enqueue refuses incoming otel.-prefixed labels. The workflow.queue attribute, the workflow.queue.wait histogram and the workflow.queue.depth gauge (ObserveQueueDepth). RunFinished() counts run ends on workflow.runs.finished by workflow.run.status; the step outcome discarded. Step spans carry workflow.node; WithTenant() records workflow.tenant on spans, never on metrics.
github.com/ubgo/workflow/admin Separate module (go 1.24, standard library and the core only): operations over any driver (runs, failures, discarded runs, retry, delete, cancel, pause, schedules, queue stats with workers per queue on a scaling driver, each refused when the driver lacks it), a closed-by-default JSON API (NewHandler with typed Identify and Authorize hooks, JSON-only writes, payloads hidden unless WithPayloads) and a queue threshold Monitor. A web UI at ui/ under the handler's mount (embedded, no third-party code, strict CSP, read-only ActionUI; Identify must accept the app's session cookie): queues, runs with filters and numbered pages, run detail, failures and discarded, schedules, pause and resume; CountRuns and count=1; schedule pause and resume routes (PauseSchedule, ResumeSchedule, OpSchedulePause); label=key=prefix*. Schedule inspect and Run now, force complete and fail, per-attempt errors (stacks with WithStacks) and captured output (WithOutputs, OutputSourceFunc, MemoryOutputs) in the API and the UI. Bulk retry and delete by filter with a dry run (RetryAll, DeleteAll), historical charts from a pluggable MetricsSource (MemoryMetrics by default), nodes, health, reschedule and cancel-step routes, and the Charts and System screens.
github.com/ubgo/workflow/sqlstore Separate module (go 1.24): Durable SQL driver on database/sql through ubgo/sqlb. SQLite, Postgres and MySQL 8 are proven by the conformance suite, with every capability but CapClosure (a Go function cannot be stored), including CapDurable and CapRunFinished. Explicit, versioned Migrate(ctx) (numbered migrations on sqlb's contrib/migrate ledger, <prefix>migrations table, engine lock; refuses a newer schema with ErrSchemaTooNew); expiring leases; one row per cron tick; a step whose lease expires more than WithMaxReclaims times (default 5) fails its attempt with ErrAbandoned, then the retry policy applies; Client.BatchStore() is a durable batch.Store; opt-in retention of finished runs (WithRetention, Prune, WithArchive); Client.EnqueueTx(ctx, tx, job) writes a run inside the app's own *sql.Tx (commits or rolls back with the app's writes; idempotency keys under a savepoint so a duplicate never aborts the caller's tx; admitted after commit within the poll interval, or at once after Client.Wake(); enqueue middleware does not run); UniqueStore is a durable middleware.UniqueStore (ADR-0005) on <prefix>unique_locks. It has MemoryUnique's semantics exactly and is atomic across processes: an upsert on Postgres and SQLite; on MySQL, delete-expired then insert, decided by the primary key. Build it from a client (Client.UniqueStore(), created by Client.Migrate) or standalone with NewUniqueStore(db, dialect, WithTablePrefix, WithClock) plus UniqueStore.Migrate, so a Hatchet app with a database gets it without running the SQL driver. Expired locks are pruned by the dispatcher or by PruneExpired. Every capability includes retry jitter and RetryAfter, cron zones and all three misfire policies; Schedule with a Name upserts by (Workflow, Name); pluggable error classifier (WithErrorClassifier, DefaultErrorClassifier, ErrorClass; connection failures match workflow.ErrUnavailable), ID generator (WithIDGenerator, IDKind), Postgres schema (WithSchema), cron dialect (WithCronParser), jitter source (WithJitterSource) and tunings (WithTxAttempts, WithTickRetention, WithHousekeepingEvery, WithPruneChunk); Retention.KeepLast; MigrationScripts exports the frozen SQL for goose, Atlas or DBA pipelines (Start needs only the tables); migration 002_schedule_misfire. DebounceStore is a durable middleware.DebounceStore on <prefix>debounce (Client.DebounceStore or standalone NewDebounceStore; its own migration list ubgo/workflow/sqlstore/debounce, 001_debounce; Swap is atomic across processes; started bursts are pruned by the dispatcher or PruneExpired). Queues and operations: named queues (WithQueues, per-queue workers and priority), durable pauses shared by every client, Runs/Retry/Delete/CancelAll, schedule list, pause, windows and tick events, Drain and WithStopWhenIdle, EnqueueMany in one transaction and BatchStore.Prune; migration 003_queues_admin; the retrying and discarded run states, Job.Chain stored with the run and edited from a running step, and WorkerScaler per process (migration 004_run_states_chain). Label index (migration 005_run_labels, <prefix>run_labels; Client.IndexLabels backfill, run by Migrate, needed once after MigrationScripts) behind CapLabelPrefix and RunQuery.Labels; WithRecycle and StopCause. Overlap, trigger-now and last fire, WithEnqueueOnly, WithErrorHistory, force, shutdown requeue proven by the suite (migration 006_schedule_history); wake-ups between clients (WithNotifier, pgnotify on Postgres LISTEN/NOTIFY); OutputStore for captured output (001_output). Also: tenants, results, progress, snooze, heartbeats, run timeouts, node identity (Nodes, Health, WithNodeID, WithSchedulerOnly, WithoutScheduler), Reschedule, CancelStep, due-time queries, FIFO per key, queue weights, lifecycle events and WithPayloadLimit (migrations 007_leader_archive, 008_run_info_nodes, 009_schedule_tenant); WithLeaderElection/WithElector for one cron fire node, WithDatabaseClock, WithLeaseSkew and ErrLeaseLost for nodes whose clocks disagree, WithFetchBatch, WithArchiveTable with ArchivedRuns (hot/cold tables), durable JournalStore and BucketStore, Schedule.Tenant.
github.com/ubgo/workflow/sqlstore/outbox Same module as sqlstore: Transactional outbox for any workflow.Client (Hatchet, memory, a middleware-wrapped client): Outbox.Write(ctx, tx, job) inserts an entry inside the app's tx; a Relay (NewRelay(target), Start/Stop/Notify/Flush) claims entries under a lease (SKIP LOCKED on Postgres/MySQL, conditional UPDATE on SQLite), enqueues each with the job's key or the entry ID as idempotency key, deletes it once accepted, and retries failures with backoff (WithBackoff, WithMaxAttempts → dead). At-least-once. Own migration list (owner ubgo/workflow/sqlstore/outbox, 001_outbox) in the same <prefix>migrations ledger; explicit Migrate. NewRelay refuses a target without CapIdempotencyKey unless AllowDuplicates; dead entries reach WithOnDead, are read by ListDead (DeadQuery) and replayed by Requeue; WithSchema, WithIDGenerator, MigrationScripts.
github.com/ubgo/workflow/sqlstore/pgnotify Same module as sqlstore: a sqlstore.Notifier on Postgres LISTEN/NOTIFY (pgnotify.New(db)), so an enqueue wakes idle workers on every machine at once; Listen needs the pgx stdlib driver (ErrDriver otherwise).
github.com/ubgo/workflow/sqlstore/pgxtx Same module as sqlstore: pgxtx.Run hands EnqueueTx and outbox.Write a *sql.Tx that runs on the app's pgx.Tx (Postgres only).
github.com/ubgo/workflow/redis Separate module (go 1.24): durable Redis driver on go-redis v9. Runs claimed whole under a renewed lease with fenced writes, reclaim after a crash, cron once across clients, idempotency keys, retries with jitter, misfire policies and zones, run list and delete, overlap, trigger-now, last fire, enqueue-only, error history, shutdown requeue and force, each proven by the conformance suite; Redis declares every capability but CapClosure and CapQueueLimitGlobal: wait steps and Signal, concurrency keys (with FIFO), rate limits, named queues with weights, pauses, chains, run-finished events, retry and bulk cancel, schedule conditions and tick events, tenants, results, progress, snooze, heartbeats, run timeouts, node identity, Health, scheduler-only and worker-only processes, worker scaling, and stores for the journal, the leaky bucket, debounce and uniqueness.
github.com/ubgo/workflow/sentry Separate module (go 1.25): a workflow.Reporter on sentry-go, one event per failure with workflow, step and status tags, never the payload, labels only on opt-in.
github.com/ubgo/workflow/recorder Separate module (go 1.24, standard library and the core only): a run history the app owns, on any driver, built from enqueue middleware, step middleware and the run-finished hook. recorder.New(store, opts...) and workflow.Use(client, rec.Plugin()); the Store contract with its merge rules (Apply*), MemoryStore, two-stage Retention with an archive hook, Redactor (JSONPaths, PerWorkflow, Chain) and payload bounds, Query (sources, outcomes, search, stale) on top of workflow.RunQuery, the Reconciler (stale and missing runs, backfill of runs started past the recorder), and recordertest.RunStore and RunDriver to prove a store or a driver.
github.com/ubgo/workflow/recorder/sqlrecorder Separate module (go 1.24, database/sql only, no database driver): the recorder's store on Postgres, MySQL 8 and SQLite: runs, steps, events and labels tables under a prefix, idempotent Migrate and MigrationScripts, retried row-locked writes, Prune with the two retention stages.
github.com/ubgo/workflow/bench Separate module (go 1.26), not a library: the benchmark harness (benchmarks).

A driver declares what it can really do. Register (Hatchet) or Start (the other drivers) refuses a workflow that needs a capability the driver lacks, and a field the driver cannot honour returns an error wrapping workflow.ErrUnsupported that names it; nothing is silently dropped. The Hatchet driver's refusals are listed in hatchet/doc.go.

Example

type greet struct{ Name string }

c := memory.New()
err := c.Register(workflow.Workflow{
	Name: "greet",
	Steps: []workflow.Step{{
		Name: "say",
		Run: workflow.Define(func(ctx context.Context, in greet) (string, error) {
			return "hello " + in.Name, nil
		}),
	}},
})
// handle err

ctx, cancel := context.WithCancel(context.Background())
defer cancel() // cancelling the Start context stops the workers
if err := c.Start(ctx); err != nil {
	// handle err
}
ref, err := c.Enqueue(ctx, workflow.Job{Workflow: "greet", Payload: []byte(`{"Name":"Ada"}`)})

A runnable version is examples/quickstart (and Example in example_test.go); the Quickstart walks through it.

Development

Commands use Task.

Command What it does
task check The gate: gofmt, vet, race tests of every module (the examples module included), gomin, then the doc checks docs:wrap, docs:links, docs:snippets, docs:words and docs:drift
task test Race tests of every module in go.work (the Hatchet live suite skips without a token)
task gomin Vets each module under the Go version its go.mod declares (core, sqlstore, admin and redis 1.24, otel and sentry 1.25, hatchet, bench and examples 1.26)
task examples:test Vet and test the examples module alone (every program under examples/ and the doc checkers)
task docs:wrap Fail on hard-wrapped prose in any tracked Markdown file
task docs:links Fail when a relative link or #anchor in README.md, docs/ or examples/ does not resolve
task docs:snippets Fail when a marked code block in the docs differs from its source region in an example; task docs:snippets:sync rewrites stale blocks
task docs:words Fail when a forbidden word (the list lives in the task) appears in the Markdown under docs/ and examples/, the top-level public Markdown files or any Go file git tracks
task docs:drift docsync check (ds CLI)
task hatchet:test:live The conformance suite plus the run-finished, batch, batch coordinator, crash and parity (queues, run query, retry, schedules, drain, pause, recycle), schedule trigger and history, enqueue-only and rate-limit scope live tests against a running Hatchet engine
task redis:test Redis driver race tests against the shared stack's Redis (localhost:7012, each test under its own key prefix, deleted after); WFTEST_REDIS_URL points it at another Redis
task bench The benchmarks, one system at a time, printing a results table; not part of check (benchmarks)
task sqlstore:test:pg sqlstore race tests on SQLite and the host Postgres (database ubgo_workflow_test, each test in its own table prefix, dropped after)
task sqlstore:test:mysql sqlstore race tests on SQLite and MySQL 8 (sqlstore/compose.yml throwaway profile on 127.0.0.1:33072, removed after the run, pass or fail)

FAQ

Is ubgo/workflow ready for production?

Not yet. The API is not stable: the drivers pass the conformance suite, but expect breaking changes while the version is v0.x.

Does it need a server?

No. The memory and SQLite drivers run inside your process; the Postgres and MySQL drivers need only the database, and the Redis driver only Redis. Hatchet is an optional driver for apps that already run a Hatchet engine.

Can I switch backends later?

Yes. ubgo/workflow is one interface over several backends, including MySQL, Redis and in-memory, and an app can move between them without rewriting its handlers.

Will a cron job fire on every replica?

Not on the SQL or Redis drivers: on SQL each cron tick is one row, and on Redis a compare-and-set moves the schedule's last tick, so a schedule declared on several clients fires once per tick, which tests run on three clients. On Hatchet the engine owns cron. To also skip a tick while the previous run is still going, set Schedule.Overlap to workflow.OverlapSkip. The memory driver is one process.

Can I make a schedule fire on one chosen node?

Yes, in two ways. On the SQL driver, WithLeaderElection (or WithElector with your own lock) limits cron firing to one elected node, and WithSchedulerOnly / WithoutScheduler split processes into schedulers and workers (Redis has the same two options). To fire crons in the process and run the jobs on a durable driver, put both behind hybrid.New. See Nodes and health and Hybrid placement.

What delivery guarantee does it give?

At least once. A step whose worker dies is run again after its lease expires, so handlers should be idempotent; idempotency keys deduplicate enqueues within one backend.

What happens when my backend does not support a feature?

Register (Hatchet) or Start (the other drivers) refuses a workflow that needs a capability the driver lacks, and a field the driver cannot honour returns an error wrapping workflow.ErrUnsupported that names it. Nothing is silently dropped or emulated.

Can I enqueue a job inside my database transaction?

Yes. On the SQL driver, Client.EnqueueTx(ctx, tx, job) writes the run in your *sql.Tx, so it commits or rolls back with your writes. For any other driver, sqlstore/outbox writes an outbox entry in your transaction and a relay enqueues it after commit.

Can I send a job only if the HTTP request that made it succeeds?

Yes. Wrap your handler with workflow.DeferHTTP and enqueue through a client wrapped with workflow.DeferEnqueue(): the request's jobs are held and sent after the handler answers below 400, and dropped on an error status or a panic. It is not a transaction; when the job must follow a database commit, use EnqueueTx or the outbox. See Deferred dispatch.

Do unique jobs and debounce work across several processes?

Yes, with a shared store. middleware.Unique and Debounce keep their state in a UniqueStore and a DebounceStore, in-process by default; sqlstore.UniqueStore and sqlstore.DebounceStore make them hold across every process that shares the database, and work beside any driver, Hatchet included.

Is there a dashboard?

Yes. The admin module's handler serves a web UI (queues, runs with filters and pages, run detail, failures, schedules, pause and resume) beside its JSON API, behind your own login: it is closed until you give it Identify and Authorize, embeds its three files, and fetches nothing from the internet.

Can I enqueue a plain Go function?

Yes. workflow.EnqueueClosure runs a closure in process on the memory driver; for work that must survive a restart, register the function under a name with workflow.NewFunc and enqueue it with a JSON argument, which runs on every driver.

Do workers on Postgres pick up new jobs at once?

With sqlstore.WithNotifier(pgnotify.New(db)), yes: an enqueue sends a Postgres NOTIFY and idle workers on other machines start at once (measured p50 38 ms against 591 ms with a one-second poll). Polling stays as the fallback, so a lost notification only delays work.

Can web servers enqueue without running workers?

Yes. A client built with sqlstore.WithEnqueueOnly() (or the Redis driver's) enqueues and reads status without Start, and runs nothing; worker processes over the same database run the work.

How do I get told when jobs fail?

middleware.AlertPlugin calls your own sender (Slack, email, a pager) on final failures, deduplicated per workflow and step; middleware.Report hands failures to an error tracker, and the sentry module sends them to Sentry without the payload.

How do I test code that enqueues jobs?

Use workflowtest.Fake, a recording client that runs nothing, with assertions such as AssertEnqueued and AssertScheduled, or run the real handlers on the memory driver with the fake clock from clock.

Contributing

task check is the gate; see CONTRIBUTING.md for the rules a change is reviewed against, and CHANGELOG.md for what is in main.

Security

Report vulnerabilities privately; see SECURITY.md.

License

Apache-2.0.

Documentation

Overview

Package workflow is one API for cron schedules, delayed jobs, durable background jobs and multi-step workflows, with drivers that range from an in-process memory driver to engines such as Hatchet.

The package holds the contract only: the Client interface every driver implements, the specs an app writes (Workflow, Step, Job, Schedule), the capability set a driver declares (Caps), the sentinel errors, and the typed helpers that sit on top of Handler. It needs nothing beyond the standard library, and deps_test.go fails the build if that changes.

Three rules hold everywhere:

  • Honest capabilities. A driver declares what it can really do. Start refuses a registered workflow that needs a capability the driver lacks (see Requires and Caps.Missing), and a call the driver cannot honour returns an error wrapping ErrUnsupported that names the field. Nothing is silently dropped.
  • Nothing runs until Start. Constructing a client opens no connections and starts no goroutines.
  • No hidden state and no wall clock in logic. Everything hangs off the value the app constructs, and time comes from a clock.Clock.

Seams a driver applies the same way, so their meaning never drifts: the retry delay (RetryPolicy.NextDelay: Backoff.Jitter, DelayFor and RetryAfter), the cron misfire policy (Misfire.Due), the zone check (IsUTC, RequiresSchedule, CheckScheduleCaps), the backend-outage marker (ErrUnavailable, Unavailable, IsUnavailable) and the reserved label prefixes (ReservedLabelPrefixes, ValidateLabels), the retry decision (RetryPolicy.Next with Attempts: Max, MaxErrors, Until), job checks (ValidateJob, CheckJob, CheckJobQueue, EffectiveQueue, EffectiveRetriesFor), the misfire plan with its dropped count (Misfire.Plan), time-of-day windows (Windows.Allows), run query filters (RunQuery.Match, ValidateRunQuery, ValidateCancelQuery), pause targets (CheckPause, PauseTarget.Holds), the labels of a fired run (FiredLabels), the status a step error ends a run with (EndStatus), and a chain's next job and edits (NextInChain, ChainEdits).

Optional interfaces beside Client, each declared by a capability: RunNotifier, RunQuerier, RunRetrier, RunDeleter, BulkCanceller, Pauser, TickNotifier, ScheduleLister, SchedulePauser, Drainer, WorkerScaler and ClosureEnqueuer. A client wrapper embeds Forward so it never hides them. RunInline runs a workflow in the caller's goroutine with no client at all.

Driver-independent pieces built on Client: Deferred, DeferEnqueue and DeferHTTP hold enqueues until a request succeeds; Balancer, ByDepth, ByWait and BalanceLoop move workers between queues on a WorkerScaler; EnqueueClosure runs a Go function on an in-process driver.

Example

Example is the README's example, kept here so it is compiled and run.

package main

import (
	"context"
	"fmt"

	"github.com/ubgo/workflow"
	"github.com/ubgo/workflow/memory"
)

func main() {
	type greet struct{ Name string }
	done := make(chan string, 1)

	c := memory.New()
	err := c.Register(workflow.Workflow{
		Name: "greet",
		Steps: []workflow.Step{{
			Name: "say",
			Run: workflow.Define(func(ctx context.Context, in greet) (string, error) {
				done <- "hello " + in.Name
				return "ok", nil
			}),
		}},
	})
	if err != nil {
		panic(err)
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel() // stops the workers
	if err := c.Start(ctx); err != nil {
		panic(err)
	}
	if _, err := c.Enqueue(ctx, workflow.Job{Workflow: "greet", Payload: []byte(`{"Name":"Ada"}`)}); err != nil {
		panic(err)
	}
	fmt.Println(<-done)
}
Output:
hello Ada

Index

Examples

Constants

View Source
const (
	// LabelParentRun is the RunID of the run whose step spawned the job.
	LabelParentRun = LabelPrefixInternal + "parent-run"

	// LabelParentStep is the name of that step.
	LabelParentStep = LabelPrefixInternal + "parent-step"
)

Labels Spawn writes on every job it enqueues, so a run listing can find the children of a run. Spawn sets them itself, overwriting any value a job carried under the same key, and enqueues under AllowReservedLabels for these two keys only: they are refused on a public Enqueue unless the owner opts in (CheckJobLabels, on drivers that enforce it), so they cannot arrive by accident or from untrusted input. Like every reserved label they are a guard, not a sandbox (see AllowReservedLabels).

View Source
const (
	// LabelPrefixInternal is owned by the middleware package (unique
	// locks, claim-check references).
	LabelPrefixInternal = "ubgo.workflow/"

	// LabelPrefixBatch is owned by the batch package (batch ID and job
	// index).
	LabelPrefixBatch = "workflow.batch."

	// LabelPrefixOTel is owned by the otel package (trace context carried
	// from Enqueue to the step).
	LabelPrefixOTel = "otel."
)

Reserved label prefixes. The library's own packages write labels under these prefixes onto the jobs they pass down (unique keys, claim-check references, batch membership, trace context), so an app label under one of them would collide with, or be removed by, the package that owns it. The three spellings differ for historical reasons and are kept as they are: runs in flight carry them, and renaming one would break those runs.

View Source
const (
	// DefaultRunsLimit is the page size of a RunQuery whose Limit is zero.
	DefaultRunsLimit = 100

	// MaxRunsLimit is the largest Limit a RunQuery may ask for, so one
	// call cannot load an unbounded table into memory.
	MaxRunsLimit = 1000
)

Page sizes of a run query.

View Source
const (
	// LabelSource holds the run's Source.
	LabelSource = LabelPrefixInternal + "source"

	// LabelSourceRef holds a free-form reference within the source: a
	// webhook delivery ID, a chat thread, the user who pressed a button.
	LabelSourceRef = LabelPrefixInternal + "source-ref"
)

Trigger source: what started a run (a schedule, an operator, a parent run, an app's webhook handler). It is plain run metadata kept in two reserved labels, so every driver stores and returns it with no change, and a run recorder or an admin view can filter "fired at 2 am" apart from "an operator pressed Run now" or "a Slack message asked for it".

Why labels and not a ctx value: metadata a caller must remember to put on a ctx gets forgotten, and the run then silently lacks it. A label travels on the Job, which every driver stores, and RequireSource (in the middleware package) can refuse a job that has none.

Why reserved yet app-settable: the keys sit under LabelPrefixInternal so no app label collides with them, but unlike the other reserved keys an app is expected to set them (WithSource), so CheckJobLabels and ValidateLabels let these two through when their values are valid. A source is provenance for people and filters, never an authorization input: anything that can enqueue can claim any source.

View Source
const (
	// MaxSourceLen is the longest Source, in bytes.
	MaxSourceLen = 64

	// MaxSourceRefLen is the longest source reference, in bytes.
	MaxSourceRefLen = 256
)

Bounds of a source and its reference, so a label value stays an index key, not a payload.

View Source
const (
	// MaxStepEventMessage is the longest StepEvent.Message, in bytes.
	MaxStepEventMessage = 1 << 10

	// MaxStepEventAttrs is the most attributes one StepEvent keeps.
	MaxStepEventAttrs = 32

	// MaxStepEventValue is the longest attribute value, in bytes.
	MaxStepEventValue = 1 << 10
)

Bounds of a step event, so a step that records events in a loop cannot grow what a sink stores without limit. RecordEvent cuts a longer message or value and keeps at most MaxStepEventAttrs attributes.

View Source
const (
	TickReasonWhen   = "when"   // Workflow.ScheduleWhen returned false
	TickReasonWindow = "window" // the tick fell outside the schedule's Windows
	TickReasonPaused = "paused" // the schedule was paused (SchedulePauser)

)

The reasons of a TickSkipped.

View Source
const ChoiceDecideSuffix = ".decide"

ChoiceDecideSuffix is appended to a Choice's name to name its decide step; each branch step is named "<choice>.<branch>", and the join step carries the Choice's own name, so later steps list the Choice's name in After as they would any step.

View Source
const ClosureStep = "closure"

ClosureStep is the name of a closure run's one step.

View Source
const ClosureWorkflow = "ubgo.workflow.closure"

ClosureWorkflow is the workflow name a closure run carries in RunInfo, RunEvent and StepInfo, so run listings, run-finished observers and middleware see closures as runs of one name. Validate refuses it as the name of a registered workflow. The closure's own name is in the LabelClosure label.

View Source
const DefaultAwaitPoll = 500 * time.Millisecond

DefaultAwaitPoll is how often Await asks Status when no AwaitOptions.Poll is given.

View Source
const DefaultBalanceEvery = 5 * time.Second

DefaultBalanceEvery is how often a BalanceLoop with no Every runs.

View Source
const DefaultChildPoll = 5 * time.Second

DefaultChildPoll is how long AwaitChild snoozes the parent step between looks at an unfinished child.

View Source
const DefaultErrorHistory = 20

DefaultErrorHistory is how many attempt errors a run keeps when the driver's error-history option is not set: enough to see a pattern in a run that retried a dozen times, small enough that a run retried for a week does not grow a row without bound.

View Source
const DefaultQueue = "default"

DefaultQueue is the queue of a run whose Job.Queue and Workflow.Queue are both empty. Naming it lets a QueueConfig give the default queue its own worker limit and priority.

View Source
const DefaultRecycleCheck = time.Second

DefaultRecycleCheck is Recycle.CheckEvery when it is zero.

View Source
const DefaultRetryMax = 0

DefaultRetryMax is the number of retries a step gets when no policy at any level (Job.Retry, Step.Retry, Workflow.Retry) says otherwise: none. It is the last link of the precedence chain EffectiveRetriesFor resolves, and the Max of the zero RetryPolicy.

View Source
const FuncStep = "func"

FuncStep is the name of the one step of a Func's workflow.

View Source
const LabelClosure = LabelPrefixInternal + "closure"

LabelClosure is the label holding a closure run's ClosureJob.Name. It sits under the reserved internal prefix, so app labels cannot forge it.

View Source
const LabelPayloadVersion = LabelPrefixInternal + "payload-version"

LabelPayloadVersion carries the version of a job's payload shape, set at enqueue by middleware.PayloadVersion, so a step can upcast a payload written by older code (Upcast).

View Source
const LabelSchedule = LabelPrefixInternal + "schedule"

LabelSchedule is the label drivers with CapTickEvents set on every run a schedule fires, to the schedule's ScheduleID, so a run-finished observer can tell which schedule a run came from (AttachScheduleHooks relies on it). It is under the reserved LabelPrefixInternal, so an app cannot set it by accident.

View Source
const LabelScheduleName = LabelPrefixInternal + "schedule-name"

LabelScheduleName is set beside LabelSchedule to the schedule's Name (for a declared cron, its expression), so a run-finished observer on any node can match a run to a named schedule without a lookup.

View Source
const LabelTickAt = LabelPrefixInternal + "tick-at"

LabelTickAt is the label a run fired by a schedule carries with the tick it represents (its logical time), RFC 3339 with nanoseconds in UTC. A job reads it through StepInfo.TickAt or TickTime: a run that starts late, or a misfire catch-up that fires three missed days in a row, still knows which day it is for, where the wall time it started would be wrong. For a manual trigger (ScheduleTriggerer) it is the driver's clock at the trigger. It is under the reserved LabelPrefixInternal, so an app cannot set it by accident. Drivers with CapTickTime set it on every fired run.

View Source
const LabelTrigger = LabelPrefixInternal + "trigger"

LabelTrigger marks a run a ScheduleTriggerer started by hand, with the value LabelTriggerManual, so run listings and observers can tell it from a cron fire.

View Source
const LabelTriggerManual = "manual"

LabelTriggerManual is the value of LabelTrigger.

View Source
const MaxChainJobs = 1000

MaxChainJobs bounds Job.Chain, counting every job after the first, so a chain stored with a run (one database row on a durable driver) stays a size a driver can load and copy on every step. A step that edits its chain past it gets an ErrInvalid.

View Source
const MaxErrorStack = 8 << 10

MaxErrorStack caps AttemptError.Stack in bytes, so one deep panic does not bloat a stored run. A longer stack keeps its first MaxErrorStack bytes, where the panicking frame is.

View Source
const (
	MaxNextRuns = 100
)

MaxNextRuns caps NextRunsQuery.Count, and maxNextRunsScan the ticks examined: a window that rejects almost every tick must not turn a display call into a long loop.

View Source
const MaxOutcomeNote = 1 << 10

MaxOutcomeNote is the longest note SetOutcome accepts, in bytes.

View Source
const MaxProgressData = 64 << 10

MaxProgressData bounds Progress.Data, so a step reporting progress in a loop cannot grow a run's stored record without limit.

View Source
const MaxSnooze = 30 * 24 * time.Hour

MaxSnooze bounds one snooze, so a bug that computes a huge delay cannot park a run for years. A longer delay is cut to it.

View Source
const MisfireGrace = time.Minute

MisfireGrace is how late MisfireSkip still fires a tick: a scheduler wakes a little after a tick's time even when nothing was missed, and a tick reached within this window is on time, not missed.

View Source
const ScheduleJSONVersion = 1

ScheduleJSONVersion is the version of the JSON form of a Schedule. A reader refuses any other version, and any field it does not know, so a schedule stored by a newer release is never loaded with part of its meaning (a window, a misfire policy) silently dropped.

View Source
const TickReasonOverlap = "overlap"

TickReasonOverlap is the reason of a TickSkipped that OverlapSkip held back because the schedule's previous run was unfinished.

Variables

View Source
var (
	// ErrUnsupported means the driver cannot honour a field or a
	// workflow's needs. Drivers return it wrapped in an UnsupportedError
	// naming the field, never by silently dropping the field.
	ErrUnsupported = errors.New("workflow: unsupported by this driver")

	// ErrNotFound means the workflow, run or schedule does not exist.
	ErrNotFound = errors.New("workflow: not found")

	// ErrNotStarted means a call that needs workers came before Start.
	ErrNotStarted = errors.New("workflow: client not started")

	// ErrAlreadyStarted means Start was called twice, or Register after
	// Start.
	ErrAlreadyStarted = errors.New("workflow: client already started")

	// ErrDuplicate means an idempotency key collided in a way the driver
	// reports as an error rather than by returning the existing run.
	ErrDuplicate = errors.New("workflow: duplicate idempotency key")

	// ErrInvalid means a Workflow, Job or Schedule failed validation.
	ErrInvalid = errors.New("workflow: invalid spec")

	// ErrCancelled is the error a run's steps see when it is cancelled;
	// it is never retried.
	ErrCancelled = errors.New("workflow: run cancelled")

	// ErrUnavailable means the backend could not be reached or did not
	// answer (a refused connection, a timeout talking to the engine or
	// database), as opposed to refusing the request. Drivers mark such
	// errors with Unavailable, so a caller such as a failover client can
	// tell an outage from an app bug without parsing driver messages.
	ErrUnavailable = errors.New("workflow: backend unavailable")

	// ErrDiscarded is the Err of a run a step ended with Discard. A
	// discarded run's Err matches ErrCancelled too, so code that treats
	// cancelled runs as "not failed" keeps working.
	ErrDiscarded = errors.New("workflow: run discarded")

	// ErrWrongState means the run exists but its status does not allow
	// the call: Retry of a run that did not fail, Delete of an unfinished
	// run.
	ErrWrongState = errors.New("workflow: run is in the wrong state for this call")
)

Sentinel errors. Callers dispatch on them with errors.Is, never on the message text.

View Source
var (
	// ErrTimeout matches every timeout below.
	ErrTimeout = errors.New("workflow: deadline missed")

	// ErrStepTimeout: a step attempt ran past Step.Timeout (or the
	// workflow's Timeout). The retry policy applies.
	ErrStepTimeout = errors.New("workflow: step attempt timed out")

	// ErrWaitTimeout: a wait step reached its Deadline or Timeout with no
	// matching event.
	ErrWaitTimeout = errors.New("workflow: wait expired")

	// ErrHeartbeatTimeout: an attempt went Step.HeartbeatTimeout without
	// calling Heartbeat. The retry policy applies.
	ErrHeartbeatTimeout = errors.New("workflow: heartbeat timed out")

	// ErrRunTimeout: the run went past Workflow.RunTimeout. It is never
	// retried.
	ErrRunTimeout = errors.New("workflow: run timed out")
)

The timeout errors. Each of ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout and ErrRunTimeout matches ErrTimeout too, so code reacting to "a deadline was missed" checks one sentinel (IsTimeout), and code that cares which deadline checks the specific one. Drivers with CapRunTimeout wrap every timeout they report with TimeoutError.

View Source
var ErrDeferredClosed = errors.New("workflow: deferred enqueues already flushed or discarded")

ErrDeferredClosed is returned by an enqueue made through a Deferred that was already flushed or discarded, such as a goroutine of a request handler enqueueing after the response was written. Nothing is sent.

View Source
var ErrDuplicatePlugin = errors.New("workflow: duplicate plugin name")

ErrDuplicatePlugin means two plugins handed to one Use call share a name.

View Source
var ErrForced = errors.New("workflow: run ended by an operator")

ErrForced is the context cause of a step attempt cut short because an operator forced its run to end (RunForcer), and the error a force-failed run reports (it matches ErrForced).

View Source
var ErrLocalCronStarted = errors.New("workflow: local cron already started")

ErrLocalCronStarted is returned by LocalCron.Add after Start and by a second Start: entries are fixed once the cron runs.

View Source
var ErrLoopLimit = errors.New("workflow: loop reached its iteration limit")

ErrLoopLimit means a Loop reached its iteration limit without its body saying it was done. It is returned marked Permanent: running the same loop again would reach the same limit.

View Source
var ErrNoJournal = errors.New("workflow: no journal on this context (use the Journal plugin)")

ErrNoJournal means Memo (or a helper built on it: MapEach, Loop, Spawn, RunData) was called from a ctx that carries no journal: the step's client was not built with the Journal plugin or the WithJournal middleware, or the call is not inside a step. It is returned marked Permanent, because retrying the step cannot wire a journal in.

View Source
var ErrNoSecretRef = errors.New("workflow: secret reference has no name")

ErrNoSecretRef means ResolveSecret got a SecretRef with no name.

View Source
var ErrOverlap = errors.New("workflow: the schedule's previous run is unfinished")

ErrOverlap is returned by ScheduleTriggerer.TriggerSchedule when the schedule's OverlapPolicy is OverlapSkip and its previous run is still unfinished: a manual trigger obeys the same rule as a tick.

View Source
var ErrPayloadTooLarge = errors.New("workflow: payload too large")

ErrPayloadTooLarge matches PayloadTooLargeError.

View Source
var ErrRecycle = errors.New("workflow: worker recycled")

ErrRecycle is matched by the error a client reports when it stopped itself because a Recycle limit was reached. A supervisor (systemd, a container restart policy, a process manager) then starts a fresh process, which frees whatever the old one leaked.

View Source
var ErrScheduleJSON = errors.New("workflow: invalid schedule JSON")

ErrScheduleJSON is wrapped by every error of the JSON form.

View Source
var ErrSecretInPayload = fmt.Errorf("%w: job carries a secret; pass a SecretRef instead", ErrInvalid)

ErrSecretInPayload means SecretGuard found what looks like a credential in a job's payload or labels. It wraps ErrInvalid, so errors.Is matches both. The error names where and which rule, never the value.

View Source
var ErrSnoozed = errors.New("workflow: step snoozed")

ErrSnoozed matches the error of a step that snoozed (Snooze), for code that inspects an attempt's outcome, such as a lifecycle observer.

View Source
var ErrStepCancelled = errors.New("workflow: step cancelled")

ErrStepCancelled is the cause of a step attempt's ctx when an operator cancelled that step (StepCanceller), and matches the step's error when it ends because of it.

View Source
var ErrThrottled = errors.New("workflow: upstream rate limit reached")

ErrThrottled means an upstream's rate limit has no room for the call now. UpstreamLimit.Take returns it marked with RetryAfter, so a step that returns it is retried once the bucket has room on a workflow whose RetryPolicy.DelayFor is RetryAfterDelay, and on its normal backoff otherwise.

Functions

func AllWhen

func AllWhen(conds ...func(context.Context, Tick) bool) func(context.Context, Tick) bool

AllWhen returns a Workflow.ScheduleWhen that fires a tick only when every condition does, asking them in order and stopping at the first false. Nil conditions are ignored; with none it always fires.

func AllowReservedLabels

func AllowReservedLabels(ctx context.Context, keys ...string) context.Context

AllowReservedLabels returns a ctx under which an Enqueue may carry the given reserved label keys, adding to what ctx already allows. The package that owns a key calls it around the Enqueue that carries the label it stamped (Spawn for its parent labels, middleware.Unique for its lock labels), so a label it wrote passes CheckJobLabels while the same key typed by an app, or copied in from a request, is refused.

It is a guard against labels arriving by accident or from untrusted input, not a sandbox: code in the same process that calls it can set any label it names, as it could call any other function. Only the owning package should.

func AllowReservedPrefix

func AllowReservedPrefix(ctx context.Context, prefix string) context.Context

AllowReservedPrefix is AllowReservedLabels for every key under prefix, for the package that owns a whole prefix (batch for LabelPrefixBatch, otel for LabelPrefixOTel).

func AppendToChain

func AppendToChain(ctx context.Context, jobs ...Job) error

AppendToChain puts jobs at the end of the chain of the run whose step ctx is, in the order given. It applies only if the step attempt succeeds, and is refused as PrependToChain is.

func AttachScheduleHooks

func AttachScheduleHooks(c Client, hooks ...ScheduleHooks) error

AttachScheduleHooks registers hooks on c before Start: one TickObserver when any hook needs ticks and one RunFinishedFunc when any needs run ends. It refuses, before registering anything, a hook without a Workflow (ErrInvalid) and a client lacking the notifier a hook needs (an UnsupportedError naming CapTickEvents or CapRunFinished). After Start the registrations return ErrAlreadyStarted.

func AwaitChild

func AwaitChild(ctx context.Context, client Client, name string, job Job, opts ChildOptions) ([]byte, error)

AwaitChild runs job as a child run of the calling step and returns the child's result: the parent waits for the child without holding a worker. The first call spawns the child (Spawn, so the Journal plugin is needed and a retried or snoozed step never spawns it twice); while the child is unfinished it returns Snooze(Poll), so the parent's step is put back without spending an attempt and asks again later; once the child succeeded it returns the child's result (client must implement ResultReader). A child that ended any other way fails the parent's step with a Permanent *RunFailedError, since asking again would see the same end.

The parent's driver needs CapSnooze: without it each look would spend an attempt of the parent step, which could fail before its child ends, so AwaitChild refuses with a Permanent UnsupportedError instead. client is the client the child runs on: the parent's own, or another backend (then set ChildOptions.Parent to the parent's client).

func AwaitChildren

func AwaitChildren(ctx context.Context, client Client, name string, jobs []Job, opts ChildOptions) ([][]byte, error)

AwaitChildren is AwaitChild for several children at once (multiple instances whose number is known when the step runs, results joined back in job order): it spawns them all, snoozes until every one is terminal, and returns their results in the order of jobs. The first child that did not succeed fails the step, as AwaitChild does.

func AwaitResult

func AwaitResult(ctx context.Context, c Client, ref RunRef, opts AwaitOptions) ([]byte, error)

AwaitResult waits like Await and returns the run's result: Result when the run succeeded (c must implement ResultReader, else an UnsupportedError), a *RunFailedError when it ended otherwise.

func CanForceComplete

func CanForceComplete(st Status) bool

CanForceComplete reports whether ForceComplete accepts a run in status st. Every driver decides with it.

func CanForceFail

func CanForceFail(st Status) bool

CanForceFail reports whether ForceFail accepts a run in status st.

func CheckCaps

func CheckCaps(have Caps, wf Workflow) error

CheckCaps returns an UnsupportedError naming wf when have lacks what wf needs, and nil otherwise. Drivers call it from Start.

func CheckJob

func CheckJob(have Caps, job Job) error

CheckJob returns an UnsupportedError when have lacks what job needs (RequiresJob), naming the first field responsible, and nil otherwise. Drivers call it from Enqueue after ValidateJob.

func CheckJobLabels

func CheckJobLabels(ctx context.Context, job Job) error

CheckJobLabels refuses, with an error wrapping ErrInvalid that names the key, a label of job (or of a job in its Chain) under a reserved prefix that ctx does not allow (AllowReservedLabels, AllowReservedPrefix). Every driver calls it first in its public Enqueue (and any other public entry that takes a Job from an app), never on its internal paths (a fired schedule, a chained job, a retry), which carry labels the library wrote. The source labels (LabelSource, LabelSourceRef) pass without an allowance when their values are valid (see WithSource). Keys are checked in sorted order, so the error is stable.

func CheckJobQueue

func CheckJobQueue(have Caps, wf Workflow, job Job) error

CheckJobQueue refuses a Job.Queue that differs from wf's effective queue when have lacks CapQueueOverride: an engine that binds a workflow to the workers of its queue cannot send one run elsewhere. Drivers call it from Enqueue once they have looked the workflow up.

func CheckPause

func CheckPause(have Caps, t PauseTarget) error

CheckPause validates t and refuses it with an UnsupportedError when have lacks what it needs. Drivers call it first in Pause and Resume.

func CheckPayload

func CheckPayload(job Job, limit int, source string) error

CheckPayload returns a *PayloadTooLargeError when job.Payload, or the payload of any job in its Chain, is over limit bytes, and nil otherwise or when limit <= 0. Drivers with a hard cap call it from Enqueue, before anything is stored.

func CheckQueueLimits

func CheckQueueLimits(have Caps, qs []QueueConfig) error

CheckQueueLimits refuses a QueueConfig list that sets Limit when have lacks CapQueueLimitGlobal, so a driver that cannot count attempts across clients says so at Start instead of silently enforcing only a per-client cap. Drivers call it next to ValidateQueues.

func CheckRunQuery

func CheckRunQuery(have Caps, q RunQuery) error

CheckRunQuery refuses what q asks for that have lacks: LabelPrefixes without CapLabelPrefix, Tenant without CapTenant, DueFrom or DueTo without CapDueQuery, as an *UnsupportedError (matches ErrUnsupported) naming the field. Every RunQuerier and BulkCanceller calls it after ValidateRunQuery, so no driver silently drops a filter.

func CheckScheduleCaps

func CheckScheduleCaps(have Caps, s Schedule) error

CheckScheduleCaps returns an UnsupportedError when have lacks what s needs, and nil otherwise. Its Field names the cause ("Schedule", "Schedule.Location", "Schedule.Windows", "Schedule.Overlap", "Schedule.Seconds", "Schedule.Active", "Schedule.KeepRuns" or "Schedule.Misfire") and its Reason the workflow. Drivers call it from Schedule, after ValidateSchedule.

func DecodePayload

func DecodePayload[T any](raw []byte) (T, error)

DecodePayload decodes a JSON payload (a step input, RunInfo.Payload, a result) into a T. Empty input decodes to the zero T. A decode failure is Permanent: retrying the same bytes cannot succeed.

func DefaultNodeID

func DefaultNodeID() string

DefaultNodeID is the node identity a driver uses when the app gives none: the host name and the process ID ("web-1:4182"), so two processes on one host are told apart. An app on a platform whose host names are meaningless (containers) passes its own through the driver's node-ID option.

func DeferHTTP

func DeferHTTP(next http.Handler, opts DeferOptions) http.Handler

DeferHTTP wraps next so every request gets its own Deferred on its context: jobs enqueued during the request through a DeferEnqueue client are sent after next returns, when opts.Succeeded accepts the response status, and dropped when it does not or when next panics (the panic goes on up after the drop). The flush runs on the request's context without its cancellation, so a client that hung up after a successful response does not cancel the jobs.

func DeferSucceeded

func DeferSucceeded(status int) bool

DeferSucceeded is DeferOptions.Succeeded's default: a status below 400.

func Discard

func Discard(err error) error

Discard marks err as "drop this run": the step is not retried, OnFailure does not run, and the run ends StatusDiscarded on a driver with CapRunStates (StatusCancelled on one with only CapDiscard; see EndStatus), with an Err matching ErrDiscarded and ErrCancelled. Use it when the work no longer makes sense (the order it was about is gone), as opposed to Permanent, which fails the run so an operator sees it. On a driver without CapDiscard it behaves as Permanent: the run fails, the direction that never hides a failure. Discard(nil) is nil.

func DiscardedError

func DiscardedError(step string, err error) error

DiscardedError is the Err a driver records for a run a step discarded: it matches ErrDiscarded, ErrCancelled and the step's own error under errors.Is. step names the step; err is what the step returned.

func DrainContext

func DrainContext(ctx context.Context, grace time.Duration) (context.Context, context.CancelFunc)

DrainContext returns ctx limited to a grace period: the running attempts get that long to finish, then the drain cuts the rest off. A grace of zero or less means no waiting at all (the returned ctx is already ended), never "wait forever": a shutdown that can hang is the bug a grace period exists to prevent. A tighter deadline already on ctx wins. The caller must call the returned cancel.

func DrainOutcome

func DrainOutcome(report DrainReport, ctxErr error) error

DrainOutcome is the one decision a drain makes about its result: given what was still running at the deadline and why the deadline ended, did the drain succeed? It is nil when nothing was cut off, even if ctx also ended (the work finished in the same instant), and ctxErr otherwise. Every driver's Drain returns this, so they cannot disagree.

func DrainTimeout

func DrainTimeout(ctx context.Context, d Drainer, grace time.Duration) error

DrainTimeout drains d, giving running attempts at most grace to finish (see DrainContext), so a caller does not hand-roll context.WithTimeout. It returns what d.Drain returns.

func EffectiveQueue

func EffectiveQueue(wf Workflow, job Job) string

EffectiveQueue is the queue a run of wf enqueued as job runs on: job.Queue, else wf.Queue, else DefaultQueue. Every driver calls it, so no two drivers resolve a queue differently.

func FiredLabels

func FiredLabels(labels map[string]string, id ScheduleID, name string) map[string]string

FiredLabels returns the labels of a run a schedule fires: a copy of the schedule's labels plus LabelSchedule and LabelScheduleName. Drivers with CapTickEvents build every fired job's labels with it.

func FiredLabelsAt

func FiredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string

FiredLabelsAt is FiredLabels plus LabelTickAt set to at. Drivers with CapTickTime build every fired job's labels with it.

func ForcedFailure

func ForcedFailure(reason string) error

ForcedFailure returns the error a force-failed run carries: it matches ErrForced and its message ends with reason. Drivers store its message as RunInfo.Error.

func HasStepEventSink added in v0.1.1

func HasStepEventSink(ctx context.Context) bool

HasStepEventSink reports whether any sink is bound on ctx, so a step can skip building an expensive message nobody will read.

func HeapObjectBytes

func HeapObjectBytes() uint64

HeapObjectBytes returns the bytes held by live and not yet swept heap objects (runtime/metrics), the default Recycle.Memory. It reads no process-wide state other than the runtime's own counters.

func Heartbeat

func Heartbeat(ctx context.Context, details []byte) error

Heartbeat reports from inside a step that the attempt is alive, so a long job is not mistaken for a stuck one (Step.HeartbeatTimeout) and, on a durable driver, keeps its lease. It is cheap enough to call in a loop. Outside a step, or on a driver without CapHeartbeat, it returns nil and does nothing: liveness is a hint, and a portable handler must not fail because one driver has no use for it.

func InEnvironment

func InEnvironment(current string, allowed ...string) func(context.Context, Tick) bool

InEnvironment returns a Workflow.ScheduleWhen that fires ticks only when current is one of allowed ("production"), for schedules that must not run in staging or on a laptop. The environment names are the app's: the library has no notion of one and reads nothing from the process, so the app passes its own value (a config field, an env var it already reads). With no allowed names it never fires. Skipped ticks are reported as TickSkipped with TickReasonWhen.

The decision is made once, here: the returned function ignores the tick. Combine it with other conditions through AllWhen.

func IsDiscarded

func IsDiscarded(err error) bool

IsDiscarded reports whether err, or any error it wraps, was marked by Discard.

func IsPermanent

func IsPermanent(err error) bool

IsPermanent reports whether err, or any error it wraps, was marked by Permanent.

func IsRelease

func IsRelease(err error) bool

IsRelease reports whether err carries a RetryAfter delay: a release, which counts against RetryPolicy.Max but not against MaxErrors.

func IsTimeout

func IsTimeout(err error) bool

IsTimeout reports whether err, or any error it wraps, is a missed deadline (matches ErrTimeout).

func IsUTC

func IsUTC(loc *time.Location) bool

IsUTC reports whether loc evaluates times as UTC: nil (which Workflow. CronLocation treats as UTC), time.UTC, or a zone named "UTC".

func IsUnavailable

func IsUnavailable(err error) bool

IsUnavailable reports whether err, or any error it wraps, is ErrUnavailable or was marked by Unavailable.

func LastTick

func LastTick(src TickSource, from, to time.Time, loc *time.Location) time.Time

LastTick returns the newest tick of src in (from, to], or zero when there is none. Drivers use it to fill Tick.At of a TickMisfired.

func Loop

func Loop[S any](ctx context.Context, name string, init S, limit int, body func(ctx context.Context, i int, state S) (next S, done bool, err error)) (S, error)

Loop is a structured loop (WC-21): it calls body with the iteration number and the current state until body reports done, and returns the final state. Each iteration is memoised under name and its number, so a retried step replays the finished iterations from the journal (without calling body) and resumes at the first one that did not finish.

limit bounds the iterations and is required (at least 1), so a loop is never unbounded by accident; reaching it without done returns a Permanent ErrLoopLimit. A body error is returned as it is and the iteration runs again on the next attempt. The state is stored as JSON after every iteration, so keep it small (a cursor, a counter), not the data itself. Loop needs the Journal plugin, as Memo does.

For a loop over steps of the graph rather than code (repeat a block of steps), enqueue a new run of the workflow from its last step: a run's graph is fixed when it starts.

func MapEach

func MapEach[In, Out any](ctx context.Context, name string, items []In, fn func(ctx context.Context, i int, item In) (Out, error), opts ...MapOption) ([]Out, error)

MapEach runs fn once per item and returns the results in the order of items: multiple instances whose number is known only when the step runs (WC-14), joined back into the step's result. Each item is memoised under name and its index, as Memo does, so a retried step runs again only the items that did not succeed; items must therefore keep their order across attempts.

Every item runs even when others fail, so successes are kept for the next attempt; the error is then the join of each failed item's error (labelled with its index), and the results are nil. A Permanent error from any item makes the joined error Permanent. When ctx ends, items not yet begun are skipped with ctx's error.

To run each instance as its own run instead (its own retries, its own worker, visible in run listings), enqueue them with Spawn. MapEach needs the Journal plugin, as Memo does.

func MarkDuplicate

func MarkDuplicate(ctx context.Context)

MarkDuplicate records that the Enqueue handling ctx returned an existing run. Drivers call it when an idempotency key returns the holder's run; middleware.Unique calls it when it returns the holder instead of ErrDuplicate. It does nothing when the caller did not ask.

func MarkInserted

func MarkInserted(ctx context.Context)

MarkInserted records that the Enqueue handling ctx started a new run. Drivers call it once the run is stored. The first mark wins, so a middleware that marked a duplicate is not overwritten.

func Memo

func Memo[T any](ctx context.Context, name string, fn func(ctx context.Context) (T, error)) (T, error)

Memo runs fn once per run under name and returns its result; on any later attempt of the same step of the same run (a step retry, or a run retried with RunRetrier) it returns the stored result without calling fn. This is step memoisation as a journal: code re-runs from the top and each named piece of work returns what it returned the first time.

Rules a caller keeps:

  • Names are stable across attempts and unique within one attempt of a step; a name used twice in one attempt is refused (Permanent, ErrInvalid), because the second call would silently get the first call's result.
  • An error from fn is returned and nothing is stored, so the next attempt calls fn again.
  • The result is stored as JSON. A stored value that no longer decodes into T (the type changed between deploys) is a Permanent error.
  • The guarantee is at least once: if the process dies after fn returns and before Save, or Save fails, fn runs again on the next attempt. Work that must not repeat also carries an idempotency key the remote side honours.

Memo needs the Journal plugin (or WithJournal) on the client, and returns a Permanent ErrNoJournal without it. Memo is safe to call from several goroutines of one step with different names.

func NextActive

func NextActive(s cron.Schedule, after time.Time, loc *time.Location, a Active) time.Time

NextActive is s.Next limited to a: the first fire time strictly after `after` that lies inside a, or the zero time when the period has ended (or s never fires again). Before From it starts the search just before From, so a tick exactly at From fires. Every driver computes ticks of a schedule with an Active period through it, so the bounds mean the same everywhere.

func OverlapRefusal

func OverlapRefusal(id ScheduleID, prev RunRef, st Status) error

OverlapRefusal returns the error TriggerSchedule gives when the overlap policy holds a trigger back. It matches ErrOverlap and names the schedule and the unfinished run.

func ParseCron

func ParseCron(expr string, seconds bool, custom cron.Parser) (cron.Schedule, error)

ParseCron parses a schedule's expression the way every driver must: seconds true (Schedule.Seconds, Workflow.CronSeconds) reads six fields with cron.ParseSeconds; otherwise custom parses it when set (a driver's WithCronParser dialect), else cron.Parse. The seconds flag wins over a custom dialect because it is part of the schedule's stored meaning, while a custom parser is one process's choice.

func PayloadVersionOf

func PayloadVersionOf(labels map[string]string) (int, error)

PayloadVersionOf returns the payload version a run's labels carry, and 0 when none is set (a job enqueued before versioning, which counts as the oldest version).

func PeriodKey

func PeriodKey(t time.Time, p Period, loc *time.Location) (string, error)

PeriodKey names the period of p that t falls in, on loc's wall clock: "2026-10-14", "2026-W42", "2026-10", "2026-Q4", "2026". Every instant of one period gets the same key, so a key built from it is stable across retries, nodes and late fires. loc is required and time.Local is refused, as for Schedule.Location: a period boundary depends on the zone, and the host's zone differs from machine to machine. Errors wrap ErrInvalid.

func Permanent

func Permanent(err error) error

Permanent marks err as non-retryable: the step fails now and its remaining attempts are skipped. Permanent(nil) is nil.

func PickWeighted

func PickWeighted(weights []int, r float64) int

PickWeighted returns the index of the queue a weighted pick chooses from weights (each >= 1 here; a zero or negative weight counts as 1) given r, a uniform value in [0, 1). Every driver that weights queues calls it with its injectable random source, so all of them choose the same way and a test can pin the draw. It returns -1 for no weights.

func PrependToChain

func PrependToChain(ctx context.Context, jobs ...Job) error

PrependToChain puts jobs in front of the rest of the chain of the run whose step ctx is, so jobs[0] runs next once the run succeeds. Several calls stack: the jobs of the latest call run first. It applies only if the step attempt succeeds. Outside a step, or on a driver without CapChain (an engine whose run shape is fixed at registration), it returns an UnsupportedError; start a child run instead there. A job that fails ValidateJob or the driver's CheckJob is refused at once, and so is one carrying a reserved label that ctx does not allow (CheckJobLabels): a chained job reaches the driver's internal enqueue, so this is where an app's labels are checked.

func ProcessChunks

func ProcessChunks(ctx context.Context, name string, opts ChunkOptions, fn func(ctx context.Context, offset, size int64) (handled int64, err error)) (int64, error)

ProcessChunks runs an export or import as chunks of Size items with progress (SA-05): it calls fn with each chunk's offset and size until fn handles fewer items than size (the source ran dry), reporting progress after every chunk with ReportProgress, so a page can show how far the job has got. It returns the number of items handled.

Each chunk is a Loop iteration, memoised in the journal, so a retried step resumes at the first chunk that did not finish instead of starting over; it needs the Journal plugin. fn must therefore be idempotent per chunk (an upsert, or a file part named by its offset), since a crash after fn and before the journal's save runs that chunk again.

The caller asked for progress, so a driver that keeps none is not silently ignored: the first report's UnsupportedError fails the step (Permanent). Reaching MaxChunks is a Permanent ErrLoopLimit.

func RecordEvent added in v0.1.1

func RecordEvent(ctx context.Context, msg string, attrs ...slog.Attr)

RecordEvent records msg with attrs on the current step's timeline, at slog.LevelInfo. With no sink bound it does nothing: an event is a log line, not data the caller asked to keep (contrast ReportProgress, which errors), so a step records events the same way with or without a recorder. Values are rendered with slog.Value.String after Resolve.

func RecordEventLevel added in v0.1.1

func RecordEventLevel(ctx context.Context, level slog.Level, msg string, attrs ...slog.Attr)

RecordEventLevel is RecordEvent at a chosen level (a warning in a timeline that a sink may colour or alert on).

func RegisterFunc

func RegisterFunc(client Client, f FuncDefinition) error

RegisterFunc registers f on client. A Func whose function is nil fails Validate (no steps) with ErrInvalid.

func ReportProgress

func ReportProgress(ctx context.Context, p Progress) error

ReportProgress records p as the run's progress from inside a step, so an app (an export page) can show how far a long job has got: read it back with RunInfo.Progress (RunQuerier) or the LifecycleProgress event. It validates p first (ErrInvalid). Outside a step, or on a driver that keeps no progress, it returns an UnsupportedError: unlike Heartbeat, the caller asked for data to be kept, so silently dropping it would be a lie. A handler that does not care can ignore the error.

func ReservedLabel

func ReservedLabel(key string) (prefix string, ok bool)

ReservedLabel reports whether key falls under a reserved prefix, and which.

func ReservedLabelPrefixes

func ReservedLabelPrefixes() []string

ReservedLabelPrefixes returns every label prefix the library reserves for itself, as a fresh slice.

func ResolveSecret

func ResolveSecret(ctx context.Context, r SecretResolver, ref SecretRef) ([]byte, error)

ResolveSecret returns the value ref names, from r. An empty ref is a Permanent ErrNoSecretRef; r's own errors are returned as they are, so a vault outage retries the step.

func RetryAfter

func RetryAfter(err error, d time.Duration) error

RetryAfter marks err with the wait the next retry should observe, such as the Retry-After of a 429. It is honoured only under a policy whose DelayFor reads it (RetryAfterDelay); under any other policy the error retries on the normal Backoff, on every driver. Permanent still wins: RetryAfter(Permanent(err), d) is not retried. RetryAfter(nil, d) is nil and a negative d counts as zero.

func RetryAfterDelay

func RetryAfterDelay(err error, _ int) (time.Duration, bool)

RetryAfterDelay is the stock RetryPolicy.DelayFor: the delay of the outermost RetryAfter in err's chain, or false when there is none.

func RunData

func RunData[T any](ctx context.Context, name string) (T, bool, error)

RunData returns the value SetRunData stored under name in this run, and false when none was stored. A stored value that does not decode into T is a Permanent error.

func RunOutput

func RunOutput(wf Workflow, outs map[string][]byte) []byte

RunOutput is the result of a succeeded run of wf whose step outputs are outs: with one final step (Sinks), its output unchanged; with several, a JSON object keyed by step name, built by the StepInput rule. Every driver with CapResult computes the result with it, so a one-step job's result is simply its handler's return value everywhere.

func Serve

func Serve(ctx context.Context, client Client) error

Serve starts client and blocks until it stops, so a worker's main is one call whose error says whether to exit for a restart:

if err := workflow.Serve(ctx, client); errors.Is(err, workflow.ErrRecycle) {
	os.Exit(0) // the supervisor starts a fresh process
}

It returns Start's error, the client's StopCause (a *RecycleError) when a recycle limit stopped it, and nil for every requested stop. It waits on the Drainer's Done channel; a client without CapDrain is waited on until ctx ends.

func SetOutcome added in v0.1.1

func SetOutcome(ctx context.Context, o Outcome, note string) error

SetOutcome records the run's outcome from inside a step, with an optional note for a person ("3 of 100 rows failed"). The last call in a run wins. It validates o and the note's length first (ErrInvalid). Without a setter on ctx (no run recorder, or ctx is not a step's) it returns an UnsupportedError, as ReportProgress does: the caller asked for data to be kept, so dropping it silently would be a lie. A step that sets an outcome whether or not one is recorded can ignore that error.

func SetRunData

func SetRunData(ctx context.Context, name string, v any) error

SetRunData stores v under name for the whole run (WX-01): every later step of the run, and every later attempt, reads it with RunData. It is the run scope between a step's own input and output (block scope) and the app's database (global scope). It writes through the journal, so it needs the Journal plugin and is as durable as the JournalStore; values are forgotten with the run's journal when the run finishes.

A later SetRunData of the same name replaces the value. Two steps running in parallel that set one name leave whichever wrote last, so give parallel steps their own names. v is stored as JSON; keep it small.

func ShutdownHook

func ShutdownHook(d Drainer, grace time.Duration) func(context.Context) error

ShutdownHook adapts a Drainer to the one shape every phased shutdown manager accepts: a func(context.Context) error to register in a phase. The hook drains d within grace, or sooner if the phase's own ctx ends first, and returns Drain's error (the ctx error when attempts were cut off). It is a plain func so any manager (ubgo/shutdown, errgroup-style stoppers, signal.NotifyContext loops) can call it; this package imports none of them. Register it in the phase that runs before the database and the HTTP listener close, since running attempts still use both.

func ShutdownHookReport

func ShutdownHookReport(d Drainer, grace time.Duration, onReport func(report DrainReport, detailed bool)) func(context.Context) error

ShutdownHookReport is ShutdownHook that hands the report to onReport (when non-nil) before returning, so the phase can log what was cut off. onReport also runs when the report is not detailed (see DrainTimeoutReport), with a zero report and detailed false.

func Sinks

func Sinks(wf Workflow) []string

Sinks returns the names of wf's final steps: the steps no other step lists in After, in declaration order.

func Snooze

func Snooze(d time.Duration) error

Snooze, returned by a step handler, puts the step back to run again after d without spending an attempt: the next try has the same StepInfo.Attempt, the retry policy is not consulted (a snooze is not a failure, so it spends none of Max or MaxErrors; Until keeps counting from the first attempt), it is not counted in RunInfo.Attempts nor recorded in RunInfo.Errors, and the run reports StatusWaiting meanwhile. Use it for "not yet" (a resource is not ready, an upstream limit is spent) as opposed to a failure. A negative d is zero (run again as soon as a worker is free); a d over MaxSnooze is MaxSnooze. A driver without CapSnooze treats it as an ordinary error, so it spends an attempt: check the capability when that matters.

func SnoozeDelay

func SnoozeDelay(err error) (time.Duration, bool)

SnoozeDelay reports whether err, or any error it wraps, is a Snooze, and its delay.

func StackOf

func StackOf(err error) string

StackOf returns the stack trace of the first error in err's chain that implements StackTracer, cut to MaxErrorStack bytes; empty when there is none.

func StepInput

func StepInput(payload []byte, parents []string, outs map[string][]byte) []byte

StepInput builds the input bytes a step receives, so every driver follows one rule:

  • no parents: the run's payload;
  • one parent: that parent's output, unchanged;
  • several parents: a JSON object keyed by parent name, in the order of parents, each output embedded as raw JSON when it is valid JSON and as a JSON string otherwise. A parent missing from outs is null.

A handler joining several parents decodes the object (Define does this for a struct whose fields are named after the parents).

The rule is part of the contract, not an implementation detail: middleware.DecodeStep relies on valid JSON being embedded unchanged (a sealed value is a JSON string) and on empty or missing outputs being null, so changing it means changing the codec with it.

func Supports

func Supports(c Client, op Op) bool

Supports reports whether c really does op: its OpProber's answer when it has one (a wrapper), else whether c implements the operation's interface. An unknown op is false.

func TickTime

func TickTime(labels map[string]string) (time.Time, bool)

TickTime returns the tick a run's labels name (LabelTickAt), in UTC; false when the run was not fired by a schedule, or by a driver without CapTickTime, or the label does not parse.

func TimeoutError

func TimeoutError(kind error, detail string) error

TimeoutError returns an error matching kind (ErrStepTimeout, ErrWaitTimeout, ErrHeartbeatTimeout or ErrRunTimeout) and ErrTimeout, with detail in its message, such as `step "fetch" attempt 2 after 30s`. A kind that is not one of them is still matched by kind, and by ErrTimeout. Drivers build every timeout error with it, so IsTimeout is reliable on every driver with CapRunTimeout.

func TriggeredLabels

func TriggeredLabels(labels map[string]string, id ScheduleID, name string) map[string]string

TriggeredLabels is FiredLabels plus LabelTrigger, the labels of a run TriggerSchedule starts. Drivers build a triggered job's labels with it.

func TriggeredLabelsAt

func TriggeredLabelsAt(labels map[string]string, id ScheduleID, name string, at time.Time) map[string]string

TriggeredLabelsAt is TriggeredLabels plus LabelTickAt set to at, the trigger time. Drivers with CapTickTime build a triggered job's labels with it.

func Unavailable

func Unavailable(err error) error

Unavailable marks err as "the backend could not be reached": errors.Is matches both ErrUnavailable and err itself. Unavailable(nil) is nil.

func Unsupported

func Unsupported(field, reason string) error

Unsupported returns an UnsupportedError for field.

func Upcast

func Upcast(raw []byte, from, to int, ups map[int]Upcaster) ([]byte, error)

Upcast brings raw, written at version from, up to version to by applying ups[from], ups[from+1], ... ups[to-1] in turn; ups[n] turns version n into n+1. A missing step in the chain, a from above to, or an upcaster's error is Permanent, since retrying the same bytes cannot help.

func Validate

func Validate(wf Workflow) error

Validate checks the shape of wf, independent of any driver: a name, at least one step, unique non-empty step names, exactly one kind per step, After naming existing steps and forming no cycle, and policies within their ranges. Every error wraps ErrInvalid and names the step or field. Drivers call it from Register; it does not parse cron expressions, which the driver's cron parser checks.

func ValidateCancelQuery

func ValidateCancelQuery(q RunQuery) error

ValidateCancelQuery checks q for CancelAll: ValidateRunQuery, no terminal status, and no Limit or Cursor (a bulk cancel covers every match, so paging it would be ambiguous). Errors wrap ErrInvalid.

func ValidateClosure

func ValidateClosure(c ClosureJob) error

ValidateClosure checks c: a Name, a Fn, a valid queue and retry policy, a Timeout that is not negative, and no label under a reserved prefix. Errors wrap ErrInvalid.

func ValidateJob

func ValidateJob(job Job) error

ValidateJob checks the shape of a job, independent of any driver and of the workflow it names: a workflow name, a valid Queue when one is set, and a Retry override within range that carries no functions (a durable driver stores the job and cannot store a NoRetryOn or DelayFor). Errors wrap ErrInvalid. Drivers call it first in Enqueue. It does not check labels, which need the caller's ctx: the library's own packages set reserved labels on the jobs they pass down, so a driver's public Enqueue also calls CheckJobLabels(ctx, job).

func ValidateLabels

func ValidateLabels(labels map[string]string) error

ValidateLabels refuses app labels under a reserved prefix with an error wrapping ErrInvalid that names the key and the prefix, instead of letting the owning package drop or overwrite them silently. Keys are checked in sorted order, so the error is the same on every call.

The source labels (LabelSource, LabelSourceRef) are the exception: an app sets them (WithSource), so they pass when their values are valid.

It is for labels an app wrote. The library's own packages set reserved labels on the jobs they pass down, so a driver's Enqueue calls CheckJobLabels instead, which lets those through; ValidateSchedule does (schedules are written only by apps), and each package that owns a prefix refuses incoming labels under its own.

func ValidatePauseTarget

func ValidatePauseTarget(t PauseTarget) error

ValidatePauseTarget refuses a target that sets none or more than one field, or names an invalid queue. Errors wrap ErrInvalid.

func ValidateProgress

func ValidateProgress(p Progress) error

ValidateProgress checks p: Percent within [0, 100] and Data within MaxProgressData. Errors wrap ErrInvalid.

func ValidateQueueName

func ValidateQueueName(name string) error

ValidateQueueName checks a queue name: 1 to 64 characters from letters, digits and ".", "_", ":", "-". Errors wrap ErrInvalid.

func ValidateQueues

func ValidateQueues(qs []QueueConfig) error

ValidateQueues checks a QueueConfig list: valid, unique names and Workers not negative. Errors wrap ErrInvalid. Drivers call it from their option or from Start.

func ValidateRecycle

func ValidateRecycle(r Recycle) error

ValidateRecycle refuses negative limits and a negative CheckEvery. Errors wrap ErrInvalid.

func ValidateRunQuery

func ValidateRunQuery(q RunQuery) error

ValidateRunQuery checks q: known statuses, a valid queue name when one is set, From before To when both are set, and 0 <= Limit <= MaxRunsLimit. Errors wrap ErrInvalid.

func ValidateSchedule

func ValidateSchedule(s Schedule) error

ValidateSchedule checks the fields every driver needs: a workflow, an expression, an explicit time zone (time.Local is refused), well-formed windows, a well-formed misfire policy and no label under a reserved prefix (see ValidateLabels). Errors wrap ErrInvalid. The cron expression itself is parsed by the driver.

func ValidateScheduleTarget

func ValidateScheduleTarget(t ScheduleTarget) error

ValidateScheduleTarget refuses a target with neither or both fields set. Errors wrap ErrInvalid.

func ValidateSource added in v0.1.1

func ValidateSource(s Source) error

ValidateSource checks s: non-empty, at most MaxSourceLen bytes, only lower-case letters, digits and . _ - (so it is safe as a filter value, a metric label and a URL segment). Errors wrap ErrInvalid.

func ValidateTenant

func ValidateTenant(tenant string) error

ValidateTenant checks a Job.Tenant or RunQuery.Tenant: 1 to 128 characters from letters, digits and ".", "_", ":", "-", "@". The characters are the queue-name set plus "@", so a tenant ID can be put into a key, a label or a log line without escaping. Errors wrap ErrInvalid.

func WithChainEditor

func WithChainEditor(ctx context.Context, ed ChainEditor) context.Context

WithChainEditor returns a ctx carrying ed. Drivers call it before invoking a step handler.

func WithDeferred

func WithDeferred(ctx context.Context, d *Deferred) context.Context

WithDeferred returns a ctx carrying d, so enqueues made with it through a DeferEnqueue client are held by d.

func WithHeartbeater

func WithHeartbeater(ctx context.Context, h Heartbeater) context.Context

WithHeartbeater returns a ctx carrying h. Drivers call it.

func WithOutcomeSetter added in v0.1.1

func WithOutcomeSetter(ctx context.Context, s OutcomeSetter) context.Context

WithOutcomeSetter returns a ctx carrying s. A run recorder's step middleware calls it.

func WithProgressReporter

func WithProgressReporter(ctx context.Context, r ProgressReporter) context.Context

WithProgressReporter returns a ctx carrying r. Drivers call it.

func WithStepEventSink added in v0.1.1

func WithStepEventSink(ctx context.Context, s StepEventSink) context.Context

WithStepEventSink returns a ctx carrying s in addition to the sinks ctx already carries, so a recorder, a tracer and a log capture each bind their own and every one sees every event, in binding order. Plugins and middleware call it on the step ctx before the handler runs. A nil s returns ctx unchanged.

func WithStepInfo

func WithStepInfo(ctx context.Context, info StepInfo) context.Context

WithStepInfo returns a ctx carrying info. Drivers call it before invoking a step handler.

Types

type Active

type Active struct {
	From, Until time.Time
}

Active is the period a schedule is live: ticks t with From <= t < Until fire, and ticks outside it are not ticks at all (no TickSkipped, no misfire catch-up across the time before From). A zero From means no start and a zero Until no end; an Until at or before From is refused, since it would mean a schedule that never fires.

func (Active) Contains

func (a Active) Contains(t time.Time) bool

Contains reports whether the tick at t is inside a.

func (Active) Ended

func (a Active) Ended(after time.Time) bool

Ended reports whether no tick after `after` can be inside a, so a driver can stop evaluating the schedule.

func (Active) IsZero

func (a Active) IsZero() bool

IsZero reports whether a limits nothing.

type AttemptError

type AttemptError struct {
	// Step is the step whose attempt failed.
	Step string

	// Attempt is the attempt number within that step, from 1.
	Attempt int

	// Error is the attempt's error message. It is a string because an
	// engine stores only the message.
	Error string

	// At is when the attempt ended, on the driver's clock.
	At time.Time

	// Node is the node that ran the attempt; empty on a driver without
	// CapNodeIdentity.
	Node string

	// Stack is the error's stack trace when it carries one (StackOf);
	// empty otherwise.
	Stack string
}

AttemptError is one failed step attempt in a run's error history (RunInfo.Errors). Drivers with CapErrorHistory record one for every attempt that returned an error, oldest first.

func AppendAttemptError

func AppendAttemptError(hist []AttemptError, e AttemptError, max int) []AttemptError

AppendAttemptError appends e to hist and keeps the newest max entries, dropping the oldest. max <= 0 keeps none (history turned off). Every driver bounds its history with it, so the bound means the same thing on each backend.

func NewAttemptError

func NewAttemptError(step string, attempt int, err error, at time.Time) AttemptError

NewAttemptError builds the history entry of a failed attempt.

type Attempts

type Attempts struct {
	// N is the number of attempts made, including the one that just
	// failed: 1 after the first.
	N int

	// Errors is how many of those attempts failed with an error that was
	// not a release (IsRelease false), including the one that just failed
	// when it is such an error.
	Errors int

	// First is when the step's first attempt started, on the driver's
	// clock. Zero disables Until.
	First time.Time
}

Attempts is what a driver knows about one step's attempts when one of them fails.

func (Attempts) Count

func (a Attempts) Count(err error) Attempts

Count returns a after one more failed attempt with err: N grows by one, and Errors too unless err is a release. Drivers keep an Attempts per step and call it on every failure, so no two drivers count differently.

type AwaitOptions

type AwaitOptions struct {
	// Poll is the wait between Status calls; zero means DefaultAwaitPoll.
	Poll time.Duration

	// Sleep waits d or until ctx ends; nil means a real timer. A test
	// passes one driven by a fake clock.
	Sleep func(ctx context.Context, d time.Duration) error
}

AwaitOptions tunes Await.

type Backoff

type Backoff struct {
	Initial time.Duration
	Factor  float64 // below 1 is treated as 1 (constant backoff)
	Max     time.Duration

	// Jitter spreads retries that failed together (an outage failing
	// thousands of runs at once) so they do not all retry at the same
	// instant: a delay d becomes a uniform value in [d*(1-Jitter), d].
	// It is a fraction in [0, 1]; zero means no jitter (the exact delay,
	// as before). Needs CapRetryJitter.
	Jitter float64
}

Backoff is an exponential wait: Initial, then times Factor per attempt, capped at Max, optionally shortened by a random Jitter.

func (Backoff) Delay

func (b Backoff) Delay(n int) time.Duration

Delay returns the wait before retry number n (1 for the first retry).

func (Backoff) JitteredDelay

func (b Backoff) JitteredDelay(n int, u float64) time.Duration

JitteredDelay is Delay(n) shortened by Jitter for the uniform draw u in [0, 1): u = 0 gives the full delay, u near 1 the shortest. The draw is an argument, not read here, so the result is a pure function a test can pin.

type BalanceConfig

type BalanceConfig struct {
	// Total is the worker budget shared by the queues. Zero means the
	// sum of their current Workers, so balancing moves workers between
	// queues without adding any. It is raised to the sum of the minimums
	// when lower.
	Total int

	// Bounds holds per-queue bounds by queue name.
	Bounds map[string]QueueBounds

	// Default bounds a queue absent from Bounds.
	Default QueueBounds

	// MaxShift caps how far one Balance call moves a queue's count, up or
	// down, so a burst does not swing every worker at once. Zero means no
	// cap. With a cap the counts can briefly sum to more or less than
	// Total while they converge.
	MaxShift int
}

BalanceConfig is the shared configuration of the built-in strategies.

type BalanceLoop

type BalanceLoop struct {
	// Scaler is the client to balance (a client declaring
	// CapWorkerScale).
	Scaler WorkerScaler

	// Balancer decides the counts.
	Balancer Balancer

	// Every is the interval; zero means DefaultBalanceEvery.
	Every time.Duration

	// Clock drives the interval; nil means clock.Real.
	Clock clock.Clock

	// OnError receives an error of one round; the loop goes on. Nil logs
	// it with Logger (nil: slog.Default()).
	OnError func(error)
	Logger  *slog.Logger
}

BalanceLoop applies a Balancer to a WorkerScaler on an interval. Balancing is off unless an app runs one: no driver balances on its own. Run it on the client's Start ctx so it stops with the client.

func (BalanceLoop) Once

func (l BalanceLoop) Once(ctx context.Context) error

Once runs one round: it reads the loads, asks the Balancer and sets every count that changed. It returns every failure joined; a count it could not set does not stop the others.

func (BalanceLoop) Run

func (l BalanceLoop) Run(ctx context.Context) error

Run calls Once every Every until ctx ends, and returns nil then. It returns at once with an error when Scaler or Balancer is missing.

type Balancer

type Balancer interface {
	Balance(loads []QueueLoad) map[string]int
}

Balancer decides how many workers each queue should have, from the queues' loads. It returns the desired count per queue name; a queue it leaves out keeps its count. A count below one is refused by the scaler, so a Balancer never stops a queue. It must be a pure function of loads (time arrives through QueueLoad.OldestWait), so a test can call it directly. ByDepth and ByWait are the built-in strategies; any other policy (a schedule, a cost model, an external autoscaler's answer) implements this interface or BalancerFunc.

func ByDepth

func ByDepth(cfg BalanceConfig) Balancer

ByDepth balances by backlog: after each queue's minimum, the budget is shared in proportion to each queue's Depth plus Running (the work it has in hand), within each queue's maximum.

func ByWait

func ByWait(cfg BalanceConfig) Balancer

ByWait balances by latency: after each queue's minimum, the budget is shared in proportion to how long each queue's oldest attempt has waited, within each queue's maximum, so the queue falling furthest behind gets the most help.

func Proportional

func Proportional(cfg BalanceConfig, weight func(QueueLoad) float64) Balancer

Proportional is the strategy behind ByDepth and ByWait with any weight: every queue gets its minimum, then the rest of the budget is shared in proportion to weight (negative counts as zero), never past a queue's maximum; what a capped queue cannot take goes to the others. When no queue has weight the rest is shared evenly. Ties go to the queue whose name sorts first, so the result is deterministic.

type BalancerFunc

type BalancerFunc func(loads []QueueLoad) map[string]int

BalancerFunc adapts a function to Balancer.

func (BalancerFunc) Balance

func (f BalancerFunc) Balance(loads []QueueLoad) map[string]int

Balance calls f.

type Branch

type Branch struct {
	// Name identifies the branch; Decide returns it. Unique within the
	// Choice.
	Name string

	// Run is the branch's work. It receives the input the decide step
	// received, and its output becomes the Choice's output.
	Run Handler

	// Retry overrides the workflow default for this branch, as Step.Retry.
	Retry *RetryPolicy

	// Timeout caps one attempt of this branch, as Step.Timeout.
	Timeout time.Duration
}

Branch is one path of a Choice: an ordinary step body with its own retry policy and timeout.

type BucketSpec

type BucketSpec struct {
	Capacity      float64
	LeakPerSecond float64
}

BucketSpec is a leaky bucket: it holds at most Capacity units of cost and drains LeakPerSecond units every second. A call of cost c fits when the level plus c is at most Capacity. Shopify's REST limit is Capacity 40, LeakPerSecond 2 with cost 1 per call; its GraphQL limit is cost based (Capacity 1000, LeakPerSecond 50, cost per query).

type BucketStore

type BucketStore interface {
	// Take adds cost to key's level and returns zero when it fits;
	// otherwise it adds nothing and returns how long until it would fit.
	Take(ctx context.Context, key string, spec BucketSpec, cost float64, now time.Time) (wait time.Duration, err error)

	// Observe sets key's level to what the upstream reported (Shopify's
	// "32/40" call-limit header, a GraphQL throttle status), so the
	// bucket follows the upstream's own count, including calls made by
	// other apps on the same credential.
	Observe(ctx context.Context, key string, spec BucketSpec, level float64, now time.Time) error
}

BucketStore holds bucket levels per key (one key per credential or shop). MemoryBuckets keeps them in this process; a shared store (Redis, SQL) makes several workers draw from one bucket, which is what an upstream limit per credential needs once more than one process calls it. Implementations apply the leak themselves, from the now they are given, and must be safe for concurrent use; Take must check and add in one atomic step.

type BulkCanceller

type BulkCanceller interface {
	CancelAll(ctx context.Context, q RunQuery) (int, error)
}

BulkCanceller cancels every run matching a query, for example to clear a queue's backlog. A client implements it when it declares CapCancelAll.

The statuses cancelled are CancelStatuses(q): by default only runs not yet started; running and waiting runs only when listed. Each run is cancelled exactly as Cancel would cancel it (its key released, its run-finished event delivered). q must pass ValidateCancelQuery. It returns how many runs it cancelled.

type Caps

type Caps uint64

Caps is a set of capabilities a driver really has. Drivers declare it through Client.Capabilities; Start compares it with Requires for every registered workflow and refuses what cannot be honoured, so a missing feature is an error at start-up rather than a silent no-op at run time.

Caps is 64 bits and 58 are used, so a new bit is kept for a spec field that Requires, RequiresJob or RequiresSchedule must check, or for data a driver would otherwise leave silently empty. An optional interface whose call can answer for itself (RunRescheduler, StepCanceller, HealthChecker, LifecycleNotifier, ReportProgress) gets no bit: it returns an UnsupportedError when the driver cannot do it.

const (
	// CapDurable: runs, waits and sleeps survive a process restart.
	CapDurable Caps = 1 << iota
	// CapDelayedStart: Job.RunAt is honoured.
	CapDelayedStart
	// CapWaitForEvent: wait steps suspend without holding a worker until
	// a Signal or their deadline. Whether the wait survives a restart is
	// CapDurable's claim.
	CapWaitForEvent
	// CapDurableSleep: sleep steps suspend without holding a worker.
	// Whether the sleep survives a restart is CapDurable's claim.
	CapDurableSleep
	// CapConcurrencyKey: Workflow.Concurrency is enforced per key.
	CapConcurrencyKey
	// CapRateLimit: Workflow.RateLimits are enforced.
	CapRateLimit
	// CapIdempotencyKey: Job.IdempotencyKey dedups enqueues.
	CapIdempotencyKey
	// CapCron: Workflow.Cron and Client.Schedule fire.
	CapCron
	// CapCancel: Client.Cancel stops a run.
	CapCancel
	// CapRunFinished: the driver implements RunNotifier with the
	// guaranteed delivery described there (every terminal run, at least
	// once). A driver that implements RunNotifier without this capability
	// delivers best-effort only.
	CapRunFinished
	// CapRetryJitter: Backoff.Jitter spreads retry delays.
	CapRetryJitter
	// CapRetryDelay: RetryPolicy.DelayFor sets a retry's delay from the
	// error (RetryAfter).
	CapRetryDelay
	// CapMisfireSkip: Misfire{Policy: MisfireSkip} is honoured.
	CapMisfireSkip
	// CapMisfireFireOnce: Misfire{Policy: MisfireFireOnce}, with Within,
	// is honoured.
	CapMisfireFireOnce
	// CapMisfireFireAll: Misfire{Policy: MisfireFireAll}, with Within and
	// Max, is honoured.
	CapMisfireFireAll
	// CapCronZone: Workflow.CronLocation and Schedule.Location may be a
	// zone other than UTC, evaluated in that zone across DST.
	CapCronZone
	// CapQueue: Workflow.Queue and Job.Queue route runs to named queues,
	// and the driver's per-queue worker limits (QueueConfig) are honoured.
	CapQueue
	// CapQueueOverride: Job.Queue may differ from the workflow's queue.
	CapQueueOverride
	// CapJobRetry: Job.Retry replaces the step policies for one run.
	CapJobRetry
	// CapRetryUntil: RetryPolicy.Until is honoured.
	CapRetryUntil
	// CapRetryMaxErrors: RetryPolicy.MaxErrors is honoured.
	CapRetryMaxErrors
	// CapDiscard: a step returning Discard(err) ends its run cancelled,
	// not failed. Without it Discard behaves as Permanent.
	CapDiscard
	// CapRunQuery: the client implements RunQuerier.
	CapRunQuery
	// CapRetryRun: the client implements RunRetrier, re-running a failed
	// run from the step that failed.
	CapRetryRun
	// CapDeleteRun: the client implements RunDeleter.
	CapDeleteRun
	// CapCancelAll: the client implements BulkCanceller.
	CapCancelAll
	// CapPauseWorkflow: the client implements Pauser for PauseTarget.Workflow
	// and PauseTarget.All.
	CapPauseWorkflow
	// CapPauseQueue: the client implements Pauser for PauseTarget.Queue.
	CapPauseQueue
	// CapSchedulePause: the client implements SchedulePauser.
	CapSchedulePause
	// CapScheduleWhen: Workflow.ScheduleWhen is asked before every tick.
	CapScheduleWhen
	// CapScheduleWindow: Schedule.Windows and Workflow.CronWindows are
	// honoured.
	CapScheduleWindow
	// CapTickEvents: the client implements TickNotifier.
	CapTickEvents
	// CapScheduleList: the client implements ScheduleLister.
	CapScheduleList
	// CapDrain: the client implements Drainer.
	CapDrain
	// CapRunStates: the driver reports StatusRetrying while a step waits
	// on its retry backoff and StatusDiscarded for a run a step ended with
	// Discard, and fills RunInfo.NextAttempt and NextAttemptAt (with
	// CapRunQuery). Without it those moments read StatusQueued or
	// StatusRunning, and a discarded run StatusCancelled (CapDiscard) or
	// StatusFailed; see EndStatus.
	CapRunStates
	// CapChain: Job.Chain is honoured (each job enqueued when the one
	// before it succeeds) and a running step may edit its run's chain with
	// PrependToChain and AppendToChain. The rest of a chain is data stored
	// with the run, so on a durable driver it survives a crash.
	CapChain
	// CapWorkerScale: the client implements WorkerScaler, so a Balancer
	// can change its per-queue worker counts while it runs.
	CapWorkerScale
	// CapClosure: the client implements ClosureEnqueuer, running a Go
	// function as a job in this process. A closure cannot be stored or
	// sent, so only an in-process driver declares it.
	CapClosure
	// CapLabelPrefix: RunQuery.LabelPrefixes is honoured by Runs and
	// CancelAll (a label value matched by prefix).
	CapLabelPrefix
	// CapOverlapSkip: OverlapSkip on Workflow.CronOverlap and
	// Schedule.Overlap is honoured, decided from the previous run's
	// stored status (OverlapPolicy.Blocks).
	CapOverlapSkip
	// CapScheduleTrigger: the client implements ScheduleTriggerer.
	CapScheduleTrigger
	// CapScheduleHistory: ScheduleInfo.Last (last fire, last run and its
	// status) is filled, and stored with the schedule on a durable driver.
	CapScheduleHistory
	// CapEnqueueOnly: the driver has an enqueue-only mode (an option of
	// the driver's constructor) whose client enqueues, and answers
	// Status, Cancel and Signal, without Start and without running any
	// worker or scheduler, so web nodes enqueue and worker nodes run. On
	// an enqueue-only client Register is optional (a registered
	// workflow's checks apply; an unregistered one gets the job-level
	// checks only) and Start returns nil without starting anything. An
	// in-process driver cannot declare it: nothing else would ever run
	// what it enqueued.
	CapEnqueueOnly
	// CapErrorHistory: RunInfo.Errors holds the run's failed attempts
	// (AttemptError), oldest first, bounded by the driver's
	// error-history option (DefaultErrorHistory when unset).
	CapErrorHistory
	// CapShutdownRequeue: a step attempt cut short by a clean stop (the
	// Start context cancelled) is put back without spending an attempt,
	// so the next worker runs it with the same attempt number and it
	// does not count toward the retry limit or appear in the error
	// history. Only a durable driver can declare it.
	CapShutdownRequeue
	// CapForceRun: the client implements RunForcer.
	CapForceRun
	// CapSnooze: a step returning Snooze(d) is put back for d without
	// spending an attempt (see Snooze).
	CapSnooze
	// CapHeartbeat: Step.HeartbeatTimeout is honoured (an attempt that
	// goes that long without Heartbeat fails with ErrHeartbeatTimeout),
	// and Heartbeat from a step reaches the driver (a durable driver
	// extends the attempt's lease).
	CapHeartbeat
	// CapTenant: Job.Tenant is stored with the run and reported on
	// StepInfo.Tenant, RunInfo.Tenant and LifecycleEvent.Tenant, and
	// RunQuery.Tenant filters by it.
	CapTenant
	// CapRunTimeout: Workflow.RunTimeout ends a run that runs too long,
	// Workflow.OnTimeout runs when a run fails because of a timeout, and
	// every timeout error matches ErrTimeout.
	CapRunTimeout
	// CapNodeIdentity: StepInfo.Node, AttemptError.Node and
	// RunInfo.AttemptedBy name the node that ran each attempt, and the
	// client implements NodeLister.
	CapNodeIdentity
	// CapKeyFIFO: Concurrency.FIFO is honoured: the queued runs of one
	// concurrency key start in enqueue order (Job.Priority aside).
	CapKeyFIFO
	// CapDueQuery: RunQuery.DueFrom and DueTo are honoured and
	// RunInfo.RunAt is filled.
	CapDueQuery
	// CapResult: the client implements ResultReader and fills
	// RunInfo.Output for a succeeded run.
	CapResult
	// CapCronSeconds: Schedule.Seconds and Workflow.CronSeconds (a
	// six-field expression, seconds first) are honoured.
	CapCronSeconds
	// CapScheduleActive: Schedule.Active and Workflow.CronActive (a start
	// and end instant) are honoured.
	CapScheduleActive
	// CapScheduleKeep: Schedule.KeepRuns and Workflow.CronKeepRuns (keep
	// only the newest N finished runs of a schedule) are honoured.
	CapScheduleKeep
	// CapTickTime: every run a schedule fires carries the tick it stands
	// for in LabelTickAt (TickTimeOf reads it), not only the wall time it
	// started.
	CapTickTime
	// CapQueueLimitGlobal: QueueConfig.Limit caps the in-flight step
	// attempts of a queue across every client sharing the store, not only
	// this client's workers (QueueConfig.Workers stays per client).
	CapQueueLimitGlobal
	// CapDiscardExhausted: RetryPolicy.DiscardOnExhaust ends a run whose
	// retries ran out StatusDiscarded instead of StatusFailed.
	CapDiscardExhausted
	// CapDrainReport: Drain reports the runs it cut off (DrainReport).
	CapDrainReport
)

The capabilities.

func Requires

func Requires(wf Workflow) Caps

Requires computes the capabilities a driver needs to run wf as written. Retry needs come from each Run step's effective policy, so a workflow default every step overrides asks for nothing. Durability is not inferred: a workflow reads the same on any driver, and the app chooses a durable driver when it needs one.

func RequiresJob

func RequiresJob(job Job) Caps

RequiresJob computes the capabilities a driver needs to honour job as written: a delayed start, an idempotency key, a queue, a tenant, a per-dispatch retry policy (with what that policy needs) and a chain (CapChain plus what every chained job needs, since the same driver enqueues them). A driver checks it in Enqueue and refuses with an UnsupportedError rather than ignoring the field.

A key counts even when the workflow's policy is IdempotencyOff. The policy is the workflow's to choose, but a key on a job is the caller's request, and a driver without the capability must say so instead of silently ignoring it: a key is never dropped without an error. It also keeps the check a pure function of the job, so a driver can refuse before looking the workflow up.

func RequiresPause

func RequiresPause(t PauseTarget) Caps

RequiresPause is the capability a target needs: CapPauseQueue for a queue, CapPauseWorkflow otherwise.

func RequiresSchedule

func RequiresSchedule(s Schedule) Caps

RequiresSchedule computes the capabilities a driver needs to honour s as written: CapCron, CapCronZone for a zone other than UTC, the capability of its misfire policy, CapScheduleWindow for windows, CapOverlapSkip for OverlapSkip, CapCronSeconds for Seconds, CapScheduleActive for an Active period, CapScheduleKeep for KeepRuns and CapTenant for Tenant.

func (Caps) Has

func (c Caps) Has(want Caps) bool

Has reports whether c holds every capability in want.

func (Caps) Missing

func (c Caps) Missing(required Caps) Caps

Missing returns the capabilities in required that c lacks; zero means c covers them all.

func (Caps) String

func (c Caps) String() string

String lists the capabilities joined by "|", such as "Durable|Cron"; the empty set is "none".

type ChainEditor

type ChainEditor interface {
	// Prepend puts jobs in front of the rest of the chain, in the order
	// given, so jobs[0] runs next once this run succeeds.
	Prepend(jobs ...Job) error
	// Append puts jobs at the end of the chain, in the order given.
	Append(jobs ...Job) error
}

ChainEditor edits the rest of the chain of the run a step belongs to. A driver with CapChain sets one on every step attempt's ctx with WithChainEditor; steps reach it through PrependToChain and AppendToChain.

Edits belong to the attempt: a driver applies them only when the attempt succeeds, so a step that fails and is retried does not add its jobs twice. ChainEdits implements this for drivers.

func ChainEditorFrom

func ChainEditorFrom(ctx context.Context) (ChainEditor, bool)

ChainEditorFrom returns the ChainEditor a driver set on ctx, and false when ctx is not a step's context or the driver lacks CapChain.

type ChainEdits

type ChainEdits struct {
	// contains filtered or unexported fields
}

ChainEdits is the ChainEditor drivers use: it checks and buffers the edits of one step attempt (the reserved-label check, which needs the step's ctx, happens in PrependToChain and AppendToChain before it), and Apply produces the run's new chain once the attempt succeeded. Make one per attempt with NewChainEdits. It is safe for concurrent use, since a step may fan out goroutines.

func NewChainEdits

func NewChainEdits(have Caps, chain []Job) *ChainEdits

NewChainEdits returns the editor of one attempt of a run whose stored chain is chain, checking each edit against have with CheckJob and the edited chain against MaxChainJobs. chain is only counted, not kept.

func (*ChainEdits) Append

func (e *ChainEdits) Append(jobs ...Job) error

Append implements ChainEditor.

func (*ChainEdits) Apply

func (e *ChainEdits) Apply(chain []Job) []Job

Apply returns chain with the buffered edits applied: the prepended jobs, then chain, then the appended jobs. chain is not modified.

func (*ChainEdits) Changed

func (e *ChainEdits) Changed() bool

Changed reports whether any edit was made.

func (*ChainEdits) Prepend

func (e *ChainEdits) Prepend(jobs ...Job) error

Prepend implements ChainEditor.

type ChildOptions

type ChildOptions struct {
	// Poll is the snooze between looks at unfinished children; zero
	// means DefaultChildPoll.
	Poll time.Duration

	// Spawn are passed to Spawn (SpawnAllowDuplicates).
	Spawn []SpawnOption

	// Parent is the client the calling step's workflow runs on, whose
	// driver must honour Snooze. Nil means the client the children run
	// on, the usual case of one backend.
	Parent Client
}

ChildOptions tunes AwaitChild and AwaitChildren.

type ChoiceSpec

type ChoiceSpec struct {
	// Name is the name of the step that carries the chosen branch's
	// output; later steps put it in After.
	Name string

	// After lists the steps the choice waits for, as Step.After. Decide
	// and the chosen branch receive their input (StepInput).
	After []string

	// Decide returns the Name of the branch to run. A name that is not a
	// branch fails the run (Permanent, ErrInvalid); an error retries the
	// decide step under the workflow's policy.
	Decide func(ctx context.Context, in []byte) (string, error)

	// Branches are the paths; at least one.
	Branches []Branch
}

ChoiceSpec is an exclusive choice (WC-04): exactly one of several branches runs, picked at run time by Decide.

type ChunkOptions

type ChunkOptions struct {
	// Total is the number of items in all, when known; it turns progress
	// into a percentage. Zero means unknown: Percent stays 0 until the
	// end and Message counts items only.
	Total int64

	// Size is the number of items per chunk; at least 1.
	Size int64

	// MaxChunks bounds the loop, so a source that never runs dry cannot
	// keep the step busy for ever. Zero means Total/Size + 1 when Total is
	// set, and is required when it is not.
	MaxChunks int
}

ChunkOptions configures ProcessChunks.

type ChunkProgress

type ChunkProgress struct {
	Done  int64 `json:"done"`
	Total int64 `json:"total,omitempty"`
}

ChunkProgress is the Data ProcessChunks puts in each Progress report, as JSON, for a page that wants the numbers rather than the message.

type Client

type Client interface {
	// Register adds a workflow definition. It validates wf (see Validate)
	// and fails with ErrAlreadyStarted after Start. Registering the same
	// name twice replaces the definition, and its declared Cron schedules
	// are replaced, never duplicated.
	Register(wf Workflow) error

	// Start checks every registered workflow against Capabilities (an error
	// wrapping ErrUnsupported names what is missing) and then starts the
	// workers. It is the only call that starts goroutines or opens
	// connections. A second Start returns ErrAlreadyStarted.
	//
	// Start returns once the workers are running; it does not block. The
	// workers live until ctx is cancelled, which is how an app stops a
	// client: no new step starts after that, and running steps see their
	// own ctx cancelled. There is no separate Stop.
	Start(ctx context.Context) error

	// Enqueue starts a run now, or later when job.RunAt is set (which
	// needs CapDelayedStart). A delayed run returns a RunRef with only
	// ScheduledID set until it starts. With an idempotency key held by an
	// unfinished run, it starts nothing and returns that run's RunRef.
	Enqueue(ctx context.Context, job Job) (RunRef, error)

	// Cancel stops a run or a delayed start. It is idempotent: cancelling
	// a finished or already cancelled run returns nil. An unknown ref
	// returns ErrNotFound.
	Cancel(ctx context.Context, ref RunRef) error

	// Status reports where a run is. An unknown ref returns ErrNotFound.
	Status(ctx context.Context, ref RunRef) (Status, error)

	// Signal delivers ev to runs whose wait step matches its name and key
	// (the fixed WaitSpec.Key, or the run's resolved KeyFrom). The
	// contract every driver keeps, and the conformance suite asserts:
	//
	//   - An event is kept from the run's enqueue: a run whose wait step
	//     begins after the Signal still takes it, so a fast reply that
	//     arrives before the run reaches its wait is not lost.
	//   - An event sent before a run was enqueued never resumes that run.
	//   - One run never takes the same event twice, so two wait steps on
	//     the same name and key need two events.
	//   - One event resumes every run waiting with that name and key, not
	//     only the first.
	//
	// A signal nobody waits for is not an error.
	Signal(ctx context.Context, ev Event) error

	// Schedule registers a recurring trigger and returns its ID, which the
	// app stores to Unschedule later. It needs CapCron, plus CapCronZone
	// for a Location other than UTC and the capability of its Misfire
	// policy (RequiresSchedule; drivers refuse with CheckScheduleCaps).
	Schedule(ctx context.Context, s Schedule) (ScheduleID, error)

	// Unschedule removes a schedule. An empty id is a no-op, so callers can
	// pass a stored ID without checking it first.
	Unschedule(ctx context.Context, id ScheduleID) error

	// Capabilities is what this driver can really do. It never changes
	// over the life of a client.
	Capabilities() Caps
}

Client is the one API an app uses; every driver implements it. App code depends on this interface only, so moving from the memory driver to an engine changes one constructor call and nothing else.

Lifecycle: Register every workflow, then Start once. Register after Start returns ErrAlreadyStarted; Enqueue, Signal and Schedule before Start return ErrNotStarted, so a forgotten Start fails loudly instead of queueing work nobody will run.

func Use

func Use(c Client, plugins ...Plugin) (Client, error)

Use applies plugins to c and returns the client to use from then on. Call it before Register and Start: step middleware reaches only workflows registered through the returned client, and observers can only be registered before Start.

Order is the order given, across plugins and within each: the first plugin's middleware is outermost, and its observers are registered first. Use checks every plugin before registering any observer on c, and refuses:

  • a nil plugin or an empty name (ErrInvalid);
  • two plugins with one name (ErrDuplicatePlugin), within this call only, since Use cannot see plugins applied by an earlier call;
  • a RunFinishedProvider when c does not implement RunNotifier (an UnsupportedError naming CapRunFinished and the plugin).

An observer registration that fails (ErrAlreadyStarted after Start) returns the error naming the plugin; observers registered before it stay registered, because RunNotifier has no way to remove one.

The returned client keeps c's Capabilities and forwards RunNotifier and every interface Forward covers to c, so wrappers stacked on it never hide the run-finished hook, run queries, pauses or drain. With no plugins Use returns c unchanged.

Example

ExampleUse adds a plugin that contributes enqueue middleware, step middleware and a run-finished observer, with one call.

package main

import (
	"context"
	"fmt"
	"sync"

	"github.com/ubgo/workflow"
	"github.com/ubgo/workflow/memory"
)

// auditLog is one ordered log several plugins can share, so a test reads
// the order of their contributions from one place.
type auditLog struct {
	mu    sync.Mutex
	lines []string
	done  chan struct{}
}

func newAuditLog(runs int) *auditLog { return &auditLog{done: make(chan struct{}, runs)} }

func (l *auditLog) record(s string) {
	l.mu.Lock()
	defer l.mu.Unlock()
	l.lines = append(l.lines, s)
}

func (l *auditLog) all() []string {
	l.mu.Lock()
	defer l.mu.Unlock()
	return append([]string(nil), l.lines...)
}

// auditPlugin uses every contribution: it records each enqueue, each step
// attempt and each run end into its log.
type auditPlugin struct {
	name string
	log  *auditLog
}

func (a *auditPlugin) Name() string { return a.name }

func (a *auditPlugin) EnqueueMiddleware() []workflow.EnqueueMiddleware {
	return []workflow.EnqueueMiddleware{func(next workflow.EnqueueFunc) workflow.EnqueueFunc {
		return func(ctx context.Context, job workflow.Job) (workflow.RunRef, error) {
			a.log.record(a.name + " enqueue " + job.Workflow)
			return next(ctx, job)
		}
	}}
}

func (a *auditPlugin) StepMiddleware() []workflow.Middleware {
	return []workflow.Middleware{func(next workflow.Handler) workflow.Handler {
		return func(ctx context.Context, in []byte) ([]byte, error) {
			a.log.record(a.name + " step")
			return next(ctx, in)
		}
	}}
}

func (a *auditPlugin) RunFinishedObservers() []workflow.RunFinishedFunc {
	return []workflow.RunFinishedFunc{func(_ context.Context, ev workflow.RunEvent) {
		a.log.record(a.name + " finished " + ev.Workflow + " " + string(ev.Status))
		a.log.done <- struct{}{}
	}}
}

func main() {
	log := newAuditLog(1)
	c, err := workflow.Use(memory.New(), &auditPlugin{name: "audit", log: log})
	if err != nil {
		panic(err)
	}
	err = c.Register(workflow.Workflow{
		Name:  "greet",
		Steps: []workflow.Step{{Name: "say", Run: func(context.Context, []byte) ([]byte, error) { return nil, nil }}},
	})
	if err != nil {
		panic(err)
	}
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()
	if err := c.Start(ctx); err != nil {
		panic(err)
	}
	if _, err := c.Enqueue(ctx, workflow.Job{Workflow: "greet"}); err != nil {
		panic(err)
	}
	<-log.done
	for _, line := range log.all() {
		fmt.Println(line)
	}
}
Output:
audit enqueue greet
audit step
audit finished greet succeeded

type ClosureEnqueuer

type ClosureEnqueuer interface {
	EnqueueClosure(ctx context.Context, c ClosureJob) (RunRef, error)
}

ClosureEnqueuer runs closures. A client implements it when it declares CapClosure. EnqueueClosure validates c (ValidateClosure), then starts a run of ClosureWorkflow whose one step (ClosureStep) calls c.Fn; the RunRef works with Status, Cancel and the run query family as any other. Before Start it returns ErrNotStarted. A closure run is never matched by an idempotency key and is never chained.

type ClosureJob

type ClosureJob struct {
	// Name says what the closure is, for logs and run listings (the
	// LabelClosure label). Required.
	Name string

	// Fn is the work. Its ctx carries StepInfo and is cancelled when the
	// run is cancelled or the client stops. Returning an error retries it
	// under Retry. Required.
	Fn func(ctx context.Context) error

	// Queue is the queue it runs on; empty means DefaultQueue.
	Queue string

	// RunAt delays the start; zero means now.
	RunAt time.Time

	// Labels are copied onto the run, beside LabelClosure.
	Labels map[string]string

	// Priority orders it among waiting runs, as Job.Priority.
	Priority int

	// Retry is the retry policy; nil means no retry. Since the closure
	// never leaves the process, NoRetryOn and DelayFor are allowed.
	Retry *RetryPolicy

	// Timeout bounds each attempt; zero means none.
	Timeout time.Duration
}

ClosureJob is a Go function run as a job in this process, for small fire-and-forget work that does not deserve a registered workflow (send a log line elsewhere, warm a cache).

A closure cannot be stored or sent: it lives only in the memory of the process that enqueued it, is lost on a restart or crash, and never runs on another machine. Anything that must survive a restart is a registered workflow with a payload. Only an in-process driver declares CapClosure; durable drivers and engines refuse closures.

type Concurrency

type Concurrency struct {
	// Key derives the bucket. A zero Key puts every run in one bucket.
	Key ConcurrencyKey

	// MaxRuns is the cap per bucket; it must be at least 1.
	MaxRuns int

	// Strategy applies when a bucket is full. Empty means
	// ConcurrencyQueueNewest.
	Strategy ConcurrencyStrategy

	// FIFO promises that the runs queued on one key start in the order
	// they were enqueued (higher Job.Priority still goes first). With
	// MaxRuns 1 that is a per-key sequence: one at a time, in insert
	// order. Only with ConcurrencyQueueNewest (the default). Needs
	// CapKeyFIFO.
	FIFO bool
}

Concurrency caps simultaneous runs per key: each distinct key value is its own bucket of MaxRuns.

type ConcurrencyKey

type ConcurrencyKey struct {
	Func func(Job) string
	Expr string
}

ConcurrencyKey derives a key from a job: the concurrency bucket, the rate-limit bucket, or a per-run wait key (WaitSpec.KeyFrom). It has two forms: a Go function (in-process drivers) or an expression in the engine's own language (Hatchet's CEL). A driver that cannot evaluate the form it is given returns an UnsupportedError naming the field. Build it with KeyFunc or KeyExpr.

func KeyExpr

func KeyExpr(expr string) ConcurrencyKey

KeyExpr returns a ConcurrencyKey written in the engine's expression language.

func KeyFunc

func KeyFunc(f func(Job) string) ConcurrencyKey

KeyFunc returns a ConcurrencyKey computed by a Go function.

func (ConcurrencyKey) IsZero

func (k ConcurrencyKey) IsZero() bool

IsZero reports whether the key is unset (one bucket for the workflow).

type ConcurrencyStrategy

type ConcurrencyStrategy string

ConcurrencyStrategy is what happens when a run arrives for a key that is already at Concurrency.MaxRuns.

const (
	ConcurrencyQueueNewest      ConcurrencyStrategy = "queue_newest"       // the new run waits
	ConcurrencyDropNewest       ConcurrencyStrategy = "drop_newest"        // the new run is dropped; see below
	ConcurrencyCancelNewest     ConcurrencyStrategy = "cancel_newest"      // the new run is cancelled; see below
	ConcurrencyCancelInProgress ConcurrencyStrategy = "cancel_in_progress" // the running one is cancelled
	ConcurrencyRoundRobin       ConcurrencyStrategy = "round_robin"        // keys take turns
)

The concurrency strategies. They map onto Hatchet's limit strategies; a driver that cannot do one returns an UnsupportedError.

Under DropNewest and CancelNewest, a run that arrives for a full bucket is still created: Enqueue returns its RunRef and a nil error, its Status is StatusCancelled, and none of its steps runs. An enqueue is therefore never an error because of concurrency, on any driver.

type DeferOptions

type DeferOptions struct {
	// Succeeded decides from the response status whether the request
	// succeeded, so its held jobs are flushed rather than discarded. Nil
	// means DeferSucceeded (any status below 400). A handler that writes
	// no status counts as 200.
	Succeeded func(status int) bool

	// OnError receives a flush that failed for some jobs. The response is
	// already written, so it cannot reach the client. Nil logs it with
	// Logger.
	OnError func(r *http.Request, err error)

	// Logger is used by the default OnError. Nil means slog.Default().
	Logger *slog.Logger
}

DeferOptions tunes DeferHTTP. The zero value is the default.

type Deferred

type Deferred struct {
	// contains filtered or unexported fields
}

Deferred holds enqueues until the work that made them is known to have succeeded: Flush sends them in order, Discard drops them. It is how a request handler enqueues "send the welcome email" without sending it when the request then fails. Put one in a ctx with WithDeferred (DeferHTTP does this per request) and enqueue through a client wrapped with DeferEnqueue.

It is not a transaction: a Flush after a database commit can still fail or the process can die between the two. When the job must be enqueued if and only if a database transaction commits, use the SQL driver's EnqueueTx or its transactional outbox instead.

A Deferred is safe for concurrent use. Use NewDeferred; it is single use: after Flush or Discard every Add returns ErrDeferredClosed.

func DeferredFrom

func DeferredFrom(ctx context.Context) (*Deferred, bool)

DeferredFrom returns the Deferred on ctx, and false when there is none.

func NewDeferred

func NewDeferred() *Deferred

NewDeferred returns an empty, open Deferred.

func (*Deferred) Add

func (d *Deferred) Add(enqueue EnqueueFunc, job Job) error

Add holds job, to be sent with enqueue by Flush. It returns ErrDeferredClosed once the Deferred was flushed or discarded. The job is copied, so the caller may reuse it.

func (*Deferred) Discard

func (d *Deferred) Discard() int

Discard drops every held job, closes d and returns how many it dropped.

func (*Deferred) Flush

func (d *Deferred) Flush(ctx context.Context) ([]RunRef, error)

Flush sends every held job, in the order held, and closes d. It sends all of them even when one fails, and returns the refs of the ones sent (the zero RunRef for a failure, so refs[i] is the i-th held job) with every failure joined. A second Flush, or a Flush after Discard, sends nothing.

func (*Deferred) Len

func (d *Deferred) Len() int

Len is the number of jobs held.

type DrainCut

type DrainCut struct {
	RunID       RunID
	Workflow    string
	Step        string
	Attempt     int
	Disposition DrainDisposition
}

DrainCut is one step attempt that was still running when a drain ran out of time. Attempt is the number the attempt had (StepInfo.Attempt); on a driver that requeues, the same number runs again, because the cut attempt was not charged.

type DrainDisposition

type DrainDisposition string

DrainDisposition says what became of an attempt a drain cut off. It is a closed set, so a caller can switch on it.

const (
	// DrainRequeued: the attempt was handed back to a durable store as
	// ready and was not charged an attempt (CapShutdownRequeue), so
	// another client runs it at once.
	DrainRequeued DrainDisposition = "requeued"
	// DrainCancelled: nothing outlives the client (an in-memory driver),
	// so the run became StatusCancelled.
	DrainCancelled DrainDisposition = "cancelled"
	// DrainEngineOwned: the engine, not this client, decides what happens
	// to the cut-off attempt (a Hatchet worker stopping mid-task: the
	// engine reclaims it after its own timeout). The driver cannot say
	// more without lying.
	DrainEngineOwned DrainDisposition = "engine"
)

type DrainReport

type DrainReport struct {
	FinishedRuns  int
	FinishedSteps int
	Cut           []DrainCut
}

DrainReport is what a drain did, for the log line a deploy script writes. FinishedSteps counts attempts that returned after the drain began and before its deadline; FinishedRuns counts runs that reached a terminal status from the drain's start until the client stopped, except runs with a cut-off attempt (a driver that cannot see runs end leaves it 0, see its doc); Cut lists the attempts still running at the deadline. A drain that waited for everything has an empty Cut.

func DrainTimeoutReport

func DrainTimeoutReport(ctx context.Context, d Drainer, grace time.Duration) (report DrainReport, detailed bool, err error)

DrainTimeoutReport is DrainTimeout that also returns the report. When d does not implement ReportingDrainer it still drains and returns an empty report with detailed false: the caller learns the report is unknown, not that nothing was cut off.

func (DrainReport) Clean

func (r DrainReport) Clean() bool

Clean reports whether the drain cut nothing off.

func (DrainReport) CutRuns

func (r DrainReport) CutRuns() int

CutRuns is the number of distinct runs with a cut-off attempt (one run can have several attempts running at once).

type DrainTracker

type DrainTracker struct {
	// contains filtered or unexported fields
}

DrainTracker is the bookkeeping every driver's drain report needs, as a mechanism a driver embeds: it knows which attempts are running, counts what finishes once a drain began, and fixes the report in two steps. The zero value is ready to use and safe for concurrent use.

Freeze runs when the drain's deadline ends, before the client's context is cancelled: the attempts still running are the cuts, and the steps that ended before are the finished steps. Seal runs once the client has stopped: it adds the runs that reached a terminal status meanwhile (a run's last result is often recorded just after its attempt returns, so counting at Freeze would miss it), minus any run with a cut-off attempt. Invariants: Begin's returned func may be called more than once and only the first call counts; Freeze and Seal are idempotent and their first results are the ones every later call returns; nothing is counted as a finished step after Freeze, so attempts the shutdown itself ends never look like finished work.

func (*DrainTracker) Begin

func (t *DrainTracker) Begin(a DrainCut) (end func())

Begin registers a running attempt and returns the func that ends it.

func (*DrainTracker) Freeze

func (t *DrainTracker) Freeze(disposition DrainDisposition) DrainReport

Freeze fixes the cuts and the finished steps: every attempt still running is a cut, with disposition. Call it when the drain's deadline ended and before the client's context is cancelled. The returned report has no FinishedRuns yet; Seal adds them.

func (*DrainTracker) RunFinished

func (t *DrainTracker) RunFinished(id RunID)

RunFinished notes that run id reached a terminal status. Before StartDrain and after Seal it does nothing.

func (*DrainTracker) Seal

func (t *DrainTracker) Seal() DrainReport

Seal completes the report once the client has stopped; call it after Freeze. A run with a cut-off attempt is not counted as finished, even if the shutdown ended it.

func (*DrainTracker) StartDrain

func (t *DrainTracker) StartDrain()

StartDrain marks that a drain began, so finishes are counted from now.

type Drainer

type Drainer interface {
	Drain(ctx context.Context) error
	Done() <-chan struct{}
}

Drainer stops a client gracefully, as before a deploy. A client implements it when it declares CapDrain.

Drain stops claiming new step attempts at once, waits until the running attempts return or ctx ends, then cancels what is left and shuts the client down exactly as cancelling the Start ctx would. Unfinished runs stay queued on a durable driver, so another client resumes them, and become StatusCancelled on an in-memory one. It returns nil when every running attempt finished, and ctx's error when it had to cut some off. Calling it again, or after the Start ctx ended, waits for the same shutdown. Before Start it returns ErrNotStarted.

Done is closed once the client has fully stopped: after Drain, after the Start ctx was cancelled, or after a driver's stop-when-idle option fired. It is the one way a process waits for its client to finish.

type EnqueueFunc

type EnqueueFunc func(ctx context.Context, job Job) (RunRef, error)

EnqueueFunc is the shape of Client.Enqueue, so middleware can wrap it.

type EnqueueMiddleware

type EnqueueMiddleware func(next EnqueueFunc) EnqueueFunc

EnqueueMiddleware wraps every enqueue: uniqueness, debouncing, routing and required-label checks live here, above any driver.

func DeferEnqueue

func DeferEnqueue() EnqueueMiddleware

DeferEnqueue is enqueue middleware that holds a job in the ctx's Deferred instead of sending it, returning the zero RunRef and nil: the run does not exist yet, so there is no ref to give. Without a Deferred on the ctx it sends the job at once, so the same client works inside and outside requests. Validation and capability checks happen when the job is flushed, by the client underneath; a caller that needs them up front, or needs the RunRef, enqueues without a Deferred on its ctx. Middleware listed after it in the chain runs at flush time.

func SecretGuard

func SecretGuard(d SecretDetector) EnqueueMiddleware

SecretGuard returns enqueue middleware that refuses a job carrying credentials, with an error matching ErrSecretInPayload (and ErrInvalid) that lists where and which rule, never the value. A payload is stored with the run, shown in admin screens and logs, and kept by retention; a secret in it outlives its rotation. Pass a SecretRef naming the secret and resolve it inside the step instead. A nil detector means NewSecretDetector(SecretRules{}).

type EnqueueMiddlewareProvider

type EnqueueMiddlewareProvider interface {
	EnqueueMiddleware() []EnqueueMiddleware
}

EnqueueMiddlewareProvider contributes enqueue middleware, run on every Enqueue through the client Use returns. The first entry sees the job first.

type EnqueueOutcome

type EnqueueOutcome struct {
	// Reported is true when something below the caller reported the
	// outcome. False means the client cannot tell (a driver without the
	// report), so Duplicate false is "unknown", not "new".
	Reported bool

	// Duplicate is true when no new run was started: the RunRef is an
	// existing run's (an idempotency key or a unique lock held by an
	// unfinished run).
	Duplicate bool
}

EnqueueOutcome is what Enqueue did beyond the RunRef it returned. A caller asks for it with WithEnqueueOutcome; drivers and enqueue middleware fill it with MarkDuplicate and MarkInserted, so it passes through any stack of wrappers without changing Enqueue's signature.

func WithEnqueueOutcome

func WithEnqueueOutcome(ctx context.Context) (context.Context, *EnqueueOutcome)

WithEnqueueOutcome returns a ctx that collects the outcome of the Enqueue it is passed to, and the outcome to read after the call:

ctx, out := workflow.WithEnqueueOutcome(ctx)
ref, err := client.Enqueue(ctx, job)
if err == nil && out.Duplicate { ... }

Use a fresh one per Enqueue: an Enqueue of a chain or a batch made with the same ctx reports its first job only.

type Event

type Event struct {
	Name    string
	Key     string
	Payload []byte
}

Event resumes runs waiting on a matching WaitSpec. Its Payload becomes the wait step's output.

type EventAttr added in v0.1.1

type EventAttr struct {
	Key   string
	Value string
}

EventAttr is one key/value pair of a StepEvent.

type Failure

type Failure struct {
	// Ref is the run.
	Ref RunRef

	// Workflow is the workflow name.
	Workflow string

	// Status is the run's end status for a final failure (StatusFailed,
	// or StatusDiscarded/StatusCancelled when the caller asked for
	// those); StatusRetrying for an attempt failure.
	Status Status

	// Step is the failed step when known; empty for a run-level report
	// from a driver that does not say which step failed.
	Step string

	// Attempt is the attempt number for an attempt failure; zero for a
	// final failure.
	Attempt int

	// Err is the error.
	Err error

	// Final is true when the run has ended, false for an attempt that
	// will be retried.
	Final bool

	// Labels are the run's labels. Receivers must not modify them; an
	// integration forwards them only on the caller's opt-in.
	Labels map[string]string

	// At is when it happened, on the observing process's clock.
	At time.Time
}

Failure is one failure handed to a Reporter or an alert sink: a run that ended badly (Final), or, when the reporter opted in, one failed step attempt that will be retried. It never carries the payload, so a third-party error tracker never receives job data.

func FailureOf

func FailureOf(ev RunEvent, discarded bool, at time.Time) (Failure, bool)

FailureOf turns a run-finished event into a final Failure, or reports false when the event is not a failure: StatusFailed always counts; StatusDiscarded counts when discarded is true; anything else does not.

type Forward

type Forward struct {
	Inner Client
}

Forward implements every optional client interface of the run query, pause, schedule, drain, scaling and closure families (RunQuerier, RunRetrier, RunDeleter, BulkCanceller, Pauser, SchedulePauser, ScheduleLister, ScheduleTriggerer, TickNotifier, Drainer, StopCauser, WorkerScaler, ClosureEnqueuer, RunForcer, RunRescheduler, StepCanceller, ResultReader, NodeLister, HealthChecker, LifecycleNotifier and PayloadLimiter) by forwarding to Inner, and refuses with an UnsupportedError naming the capability when Inner does not implement one. A client wrapper embeds it beside the Client it wraps, so stacking a wrapper never hides what the inner client can do:

type wrapper struct {
	workflow.Client
	workflow.Forward
}
w := &wrapper{Client: inner, Forward: workflow.Forward{Inner: inner}}

A wrapper that rewrites RunRefs or queries (failover's prefixed refs) implements those methods itself instead. RunNotifier is not here: the client Use returns forwards it on its own.

func (Forward) CancelAll

func (f Forward) CancelAll(ctx context.Context, q RunQuery) (int, error)

CancelAll forwards to Inner's BulkCanceller.

func (Forward) CancelStep

func (f Forward) CancelStep(ctx context.Context, ref RunRef, step string) error

CancelStep forwards to Inner's StepCanceller.

func (Forward) Delete

func (f Forward) Delete(ctx context.Context, ref RunRef) error

Delete forwards to Inner's RunDeleter.

func (Forward) Done

func (f Forward) Done() <-chan struct{}

Done forwards to Inner's Drainer. An inner client without one has no way to say it stopped, so Done returns a nil channel, which never closes; check CapDrain before waiting on it.

func (Forward) Drain

func (f Forward) Drain(ctx context.Context) error

Drain forwards to Inner's Drainer.

func (Forward) DrainWithReport

func (f Forward) DrainWithReport(ctx context.Context) (DrainReport, error)

DrainWithReport forwards to Inner's ReportingDrainer; an inner client without one returns an UnsupportedError naming CapDrainReport.

func (Forward) EnqueueClosure

func (f Forward) EnqueueClosure(ctx context.Context, c ClosureJob) (RunRef, error)

EnqueueClosure forwards to Inner's ClosureEnqueuer. Enqueue middleware of the wrapper does not run for a closure: it has no Job to rewrite.

func (Forward) ForceComplete

func (f Forward) ForceComplete(ctx context.Context, ref RunRef) error

ForceComplete forwards to Inner's RunForcer.

func (Forward) ForceFail

func (f Forward) ForceFail(ctx context.Context, ref RunRef, reason string) error

ForceFail forwards to Inner's RunForcer.

func (Forward) Health

func (f Forward) Health(ctx context.Context) HealthReport

Health forwards to Inner (CheckHealth, so an inner client without a HealthChecker reads as healthy with a check saying it cannot tell).

func (Forward) Nodes

func (f Forward) Nodes(ctx context.Context) ([]NodeInfo, error)

Nodes forwards to Inner's NodeLister.

func (Forward) OnLifecycle

func (f Forward) OnLifecycle(fn LifecycleFunc) error

OnLifecycle forwards to Inner's LifecycleNotifier.

func (Forward) OnTick

func (f Forward) OnTick(fn TickObserver) error

OnTick forwards to Inner's TickNotifier.

func (Forward) Pause

func (f Forward) Pause(ctx context.Context, t PauseTarget) error

Pause forwards to Inner's Pauser.

func (Forward) PauseSchedule

func (f Forward) PauseSchedule(ctx context.Context, t ScheduleTarget) error

PauseSchedule forwards to Inner's SchedulePauser.

func (Forward) Paused

func (f Forward) Paused(ctx context.Context) ([]PauseTarget, error)

Paused forwards to Inner's Pauser.

func (Forward) PayloadLimit

func (f Forward) PayloadLimit() int

PayloadLimit forwards to Inner's PayloadLimiter; zero (no cap) when Inner has none.

func (Forward) QueueLoads

func (f Forward) QueueLoads(ctx context.Context) ([]QueueLoad, error)

QueueLoads forwards to Inner's WorkerScaler.

func (Forward) Reschedule

func (f Forward) Reschedule(ctx context.Context, ref RunRef, at time.Time) (RunRef, error)

Reschedule forwards to Inner's RunRescheduler.

func (Forward) Result

func (f Forward) Result(ctx context.Context, ref RunRef) ([]byte, error)

Result forwards to Inner's ResultReader.

func (Forward) Resume

func (f Forward) Resume(ctx context.Context, t PauseTarget) error

Resume forwards to Inner's Pauser.

func (Forward) ResumeSchedule

func (f Forward) ResumeSchedule(ctx context.Context, t ScheduleTarget) error

ResumeSchedule forwards to Inner's SchedulePauser.

func (Forward) Retry

func (f Forward) Retry(ctx context.Context, ref RunRef) (RunRef, error)

Retry forwards to Inner's RunRetrier.

func (Forward) Runs

func (f Forward) Runs(ctx context.Context, q RunQuery) (RunPage, error)

Runs forwards to Inner's RunQuerier.

func (Forward) Schedules

func (f Forward) Schedules(ctx context.Context) ([]ScheduleInfo, error)

Schedules forwards to Inner's ScheduleLister.

func (Forward) SetWorkers

func (f Forward) SetWorkers(ctx context.Context, queue string, n int) error

SetWorkers forwards to Inner's WorkerScaler.

func (Forward) StopCause

func (f Forward) StopCause() error

StopCause forwards to Inner's StopCauser; an inner client without one never stops itself, so it returns nil.

func (Forward) Supports

func (f Forward) Supports(op Op) bool

Supports answers for Inner (see OpProber), so a wrapper embedding Forward reports only what the wrapped client really does.

func (Forward) TriggerSchedule

func (f Forward) TriggerSchedule(ctx context.Context, id ScheduleID) (RunRef, error)

TriggerSchedule forwards to Inner's ScheduleTriggerer.

type Func

type Func[In any] struct {
	// contains filtered or unexported fields
}

Func is a durable closure: a Go function registered under a name when the process starts and enqueued by that name with an encoded argument. A closure itself cannot be stored or sent to another machine, but a name and a JSON argument can, so a Func survives a restart and runs on any worker that registered the same name, on every driver (a Func is a one-step workflow underneath; no capability beyond the driver's own is needed). For a function that only ever runs in this process, the quick form is EnqueueClosure.

Build one with NewFunc, register it with RegisterFunc(client, f) (or client.Register(f.Workflow())) in every process that runs workers, and enqueue with f.Enqueue or f.Job. The argument is encoded as JSON; input that does not decode is a Permanent error.

func NewFunc

func NewFunc[In any](name string, fn func(ctx context.Context, in In) error, opts ...FuncOption) Func[In]

NewFunc names fn. The name is the workflow name runs carry in RunInfo, RunEvent and StepInfo; Validate checks it at Register. Two Funcs with one name replace each other, as two workflows do.

func (Func[In]) Enqueue

func (f Func[In]) Enqueue(ctx context.Context, client Client, in In) (RunRef, error)

Enqueue starts a run of the Func with in on client.

func (Func[In]) Job

func (f Func[In]) Job(in In) (Job, error)

Job returns the Job that runs the Func with in, for a caller that sets RunAt, Labels, Priority, IdempotencyKey or Queue before Enqueue.

func (Func[In]) Name

func (f Func[In]) Name() string

Name returns the name the Func was built with.

func (Func[In]) Workflow

func (f Func[In]) Workflow() Workflow

Workflow returns the one-step workflow that runs the Func, for client.Register. Its step (FuncStep) decodes the argument and calls the function; OnFailure runs the Catch.

type FuncConfig

type FuncConfig struct {
	// Queue is the default queue of its runs; empty means DefaultQueue.
	Queue string

	// Retry is the retry policy of its one step; nil means no retry.
	Retry *RetryPolicy

	// Timeout bounds each attempt; zero means none.
	Timeout time.Duration

	// Catch runs once when a run fails for good, with the decoded
	// argument. Nil means nothing runs. Its error is reported the way the
	// driver reports an OnFailure error.
	Catch func(ctx context.Context, in []byte) error
}

FuncConfig holds what FuncOptions set: the same knobs a ClosureJob has that belong to the definition rather than to one enqueue. RunAt, Labels, Priority, IdempotencyKey and a per-run Queue are set on the Job (see Func.Job).

type FuncDefinition

type FuncDefinition interface {
	Name() string
	Workflow() Workflow
}

FuncDefinition is what RegisterFunc accepts: a Func of any argument type.

type FuncOption

type FuncOption func(*FuncConfig)

FuncOption configures a Func.

func FuncCatch

func FuncCatch[In any](fn func(ctx context.Context, in In) error) FuncOption

FuncCatch runs fn once when a run fails for good, with the argument the run was enqueued with, decoded. An argument that does not decode is passed as the zero In.

func FuncQueue

func FuncQueue(q string) FuncOption

FuncQueue sets the Func's default queue.

func FuncRetry

func FuncRetry(p *RetryPolicy) FuncOption

FuncRetry sets the retry policy of the Func's step. Nil means no retry.

func FuncTimeout

func FuncTimeout(d time.Duration) FuncOption

FuncTimeout bounds each attempt.

type Handler

type Handler func(ctx context.Context, in []byte) (out []byte, err error)

Handler runs one step. Its input follows StepInput: the run's payload for a first step, a single parent's output unchanged, or a JSON object keyed by parent name for several parents. It returns bytes the engine stores as the step's output. Returning Permanent(err) fails the step at once, skipping the remaining retries. Handlers read time from an injected clock, never from time.Now, so a replay or a test sees the same values.

func Define

func Define[In, Out any](fn func(ctx context.Context, in In) (Out, error)) Handler

Define wraps a typed function into a Handler that decodes its input from JSON and encodes its output to JSON, so app code works with structs, not bytes. Empty input decodes to the zero In. Input that does not decode is a Permanent error: retrying the same bytes can never succeed.

func DefineVersioned

func DefineVersioned[In, Out any](current int, ups map[int]Upcaster, fn func(ctx context.Context, in In) (Out, error)) Handler

DefineVersioned is Define for a first step whose payload shape has versions: it reads the run's payload version (PayloadVersionOf on the StepInfo labels), upcasts the input to current with ups, then decodes it into In. Pair it with middleware.PayloadVersion(current), which stamps new jobs. Use it on steps that receive the run's payload (no After); a later step's input is a parent's output, which has no version.

func FailureHandler

func FailureHandler(wf Workflow, err error) Handler

FailureHandler is the handler a driver runs when a run fails for good with err: wf.OnTimeout when err is a timeout and OnTimeout is set, else wf.OnFailure (nil when neither applies). Every driver calls it, so none decides differently.

type HealthCheck

type HealthCheck struct {
	// Name says what was checked ("database", "engine", "workers").
	Name string
	// OK is the check's verdict.
	OK bool
	// Detail says why it failed, or a short fact when it passed.
	Detail string
}

HealthCheck is one check of a HealthReport.

type HealthChecker

type HealthChecker interface {
	Health(ctx context.Context) HealthReport
}

HealthChecker is an optional interface a client implements to report its health, for a liveness or readiness probe. Health must answer within ctx's deadline and must not block on a broken backend longer than that: a probe that hangs is worse than one that fails.

type HealthReport

type HealthReport struct {
	Live, Ready bool
	Checks      []HealthCheck
}

HealthReport is a client's health: Live means the process should keep running (restart it when false); Ready means it can take work now (route traffic to it, or not, when false). A client is live but not ready before Start and while its backend is unreachable.

func CheckHealth

func CheckHealth(ctx context.Context, c Client) HealthReport

CheckHealth asks c for its health. A client without HealthChecker is reported live and ready, with one check saying it cannot tell: an app on such a driver needs its own probe.

type HeartbeatFunc

type HeartbeatFunc func(ctx context.Context, details []byte) error

HeartbeatFunc adapts a function to Heartbeater.

func (HeartbeatFunc) Heartbeat

func (f HeartbeatFunc) Heartbeat(ctx context.Context, details []byte) error

Heartbeat calls f.

type Heartbeater

type Heartbeater interface {
	// Heartbeat records liveness. details is an optional progress marker
	// the driver may keep (nil is fine). An error means the beat was not
	// recorded (the backend is unreachable, or the attempt has lost its
	// lease and will be run elsewhere); a handler that gets
	// ErrCancelled should stop.
	Heartbeat(ctx context.Context, details []byte) error
}

Heartbeater receives a running attempt's heartbeats. A driver with CapHeartbeat sets one on the step's ctx (WithHeartbeater) before the Middleware chain runs: it resets the attempt's HeartbeatTimeout clock and, on a durable driver, extends the attempt's lease.

type IdempotencyPolicy

type IdempotencyPolicy string

IdempotencyPolicy says how Job.IdempotencyKey is treated.

const (
	IdempotencyOptional IdempotencyPolicy = "optional" // honoured when present (the default)
	IdempotencyRequired IdempotencyPolicy = "required" // an enqueue without a key fails
	IdempotencyOff      IdempotencyPolicy = "off"      // keys are ignored
)

The idempotency policies.

type InlineOption

type InlineOption func(*inlineConfig)

InlineOption configures RunInline.

func InlineEvents

func InlineEvents(events ...Event) InlineOption

InlineEvents supplies the events wait steps take: each wait step takes the first event not yet taken whose name and resolved key match. A wait step with no matching event fails as if it timed out. Without this option a workflow with a wait step is refused.

func InlineMiddleware

func InlineMiddleware(mw ...Middleware) InlineOption

InlineMiddleware wraps every step and OnFailure handler, first outermost, the way a driver's middleware option does. Pass the same middleware the app gives its driver to run steps exactly as they run there.

func InlineNow

func InlineNow(now func() time.Time) InlineOption

InlineNow replaces RunInline's clock, read for RetryPolicy.Until and WaitSpec.Deadline. The default is time.Now. Nil is ignored.

func InlineSleep

func InlineSleep(sleep func(ctx context.Context, d time.Duration) error) InlineOption

InlineSleep replaces how RunInline waits (retry backoffs and sleep steps). The default is a real timer that ends early with ctx. A test passes a function that advances a fake clock, or one that returns at once. Nil is ignored.

type InlineResult

type InlineResult struct {
	// Outputs holds the output of every step that succeeded (a wait
	// step's event payload, a sleep step's input), by step name.
	Outputs map[string][]byte

	// Attempts holds how many attempts each Run step made.
	Attempts map[string]int
}

InlineResult is what RunInline returns, success or failure.

func RunInline

func RunInline(ctx context.Context, wf Workflow, job Job, opts ...InlineOption) (InlineResult, error)

RunInline runs wf once, as job, in the caller's goroutine: no client, no Start, no goroutine of its own, for tests, command-line tools and one-off scripts.

Steps run one at a time in a fixed order: of the steps whose parents have all succeeded, the first in declaration order goes next. Each gets its StepInput, a StepInfo (RunID InlineRunID), its timeout, its retry policy (EffectiveRetriesFor, decided by RetryPolicy.Next, waiting through the sleeper) and the InlineMiddleware. Sleep steps sleep. Wait steps take an InlineEvents event; a workflow with a wait step and no InlineEvents option is refused with an UnsupportedError before anything runs. When a step fails for good, OnFailure runs once with that step's input, and the error is returned wrapped with the step name. A step that returns Discard, or whose policy has DiscardOnExhaust and gives up, ends the call with an error matching ErrDiscarded and ErrCancelled, without OnFailure.

Concurrency, rate limits, idempotency keys, cron, queues, priority and RunAt shape how a driver schedules many runs; a single inline call has nothing to schedule, so it ignores them. It validates wf and job first (ErrInvalid).

type Job

type Job struct {
	// Workflow is the registered workflow name.
	Workflow string

	// Payload is the input of the first steps.
	Payload []byte

	// RunAt delays the start. Zero means now. Needs CapDelayedStart.
	RunAt time.Time

	// IdempotencyKey dedups enqueues while a run holding the key has not
	// finished. Honoured per the workflow's Idempotency policy; needs
	// CapIdempotencyKey.
	IdempotencyKey string

	// Labels carry app metadata (source, trace) explicitly, so a
	// middleware can require one instead of trusting a context convention.
	Labels map[string]string

	// Tenant is the tenant the run belongs to, a first-class field rather
	// than a label convention: drivers store it, report it on
	// StepInfo.Tenant and RunInfo.Tenant, and filter by it
	// (RunQuery.Tenant). Empty means none. It must pass ValidateTenant.
	// Each chained job carries its own Tenant. Needs CapTenant.
	Tenant string

	// Priority orders runs waiting for a worker: higher goes first. Zero
	// is the default priority.
	Priority int

	// Queue overrides the workflow's queue for this run. Empty means the workflow's (EffectiveQueue). Needs
	// CapQueue, and CapQueueOverride when it differs from the workflow's
	// queue (CheckJobQueue).
	Queue string

	// Retry replaces every step's effective policy for this run (a retry
	// budget chosen per dispatch), so it wins over Step.Retry and Workflow.Retry;
	// see EffectiveRetriesFor. It is data only: NoRetryOn and DelayFor
	// must be nil (ValidateJob refuses them), because a durable driver
	// stores the job and cannot store a function. Nil means no override.
	// Needs CapJobRetry plus what the policy itself needs.
	Retry *RetryPolicy

	// Chain lists the jobs that run after this one, in order: when this
	// run succeeds the driver enqueues Chain[0] carrying Chain[1:] as its
	// own Chain (NextInChain), so each job starts only once the one before
	// it succeeded. A run that fails, is discarded or is cancelled ends
	// the chain. A running step may edit the rest of its run's chain with
	// PrependToChain and AppendToChain. The chain is data stored with the
	// run, so on a durable driver it survives a crash. Enqueue middleware
	// runs for this job only, not for the jobs the driver enqueues later.
	// At most MaxChainJobs jobs; each passes ValidateJob. Needs CapChain
	// plus what every chained job needs.
	Chain []Job
}

Job is one request to run a workflow.

func CloneJob

func CloneJob(j Job) Job

CloneJob returns a deep copy of j (Payload, Labels, Retry and Chain), so a driver or a recorder can keep a job without sharing the caller's slices and maps.

func FuncJob

func FuncJob(name string, arg any) (Job, error)

FuncJob returns the Job that runs the Func registered under name with arg encoded as JSON, for a caller that knows only the name (another service, an admin action). The driver refuses a name no process registered the way it refuses any unknown workflow.

func JobOf

func JobOf[T any](workflow string, v T) (Job, error)

JobOf builds a Job for workflow whose Payload is v encoded as JSON, the typed counterpart of filling Job.Payload by hand. Labels, RunAt and the rest are set on the returned Job as usual.

func NextInChain

func NextInChain(chain []Job) (Job, bool)

NextInChain is the job a driver enqueues when a run with this chain succeeds: chain[0], carrying its own Chain followed by chain[1:], so the rest of the chain travels with it. It returns false for an empty chain. Every driver calls it, so a chain flattens the same way everywhere. The result shares nothing with chain.

func OncePerPeriod

func OncePerPeriod(job Job, scope string, p Period, t time.Time, loc *time.Location) (Job, error)

OncePerPeriod returns job with an IdempotencyKey of "period/<scope>/<PeriodKey>", for work that must happen once per scope per period (SA-02: charge customer 42 once a month): every enqueue of the same scope in the same period, from any node and any number of schedule fires, carries the same key. It needs CapIdempotencyKey. A job that already has a key is refused (ErrInvalid) rather than overwritten.

What the key does and does not cover: a driver dedups on it while the run holding it is unfinished (Client.Enqueue). To keep a later enqueue in the same period from running again after that run succeeded, also pass the same key to the payment provider's own idempotency key (the period key alone, PeriodKey, is the stable value for that), and memoise the charge with Memo so a retried step does not charge twice.

func WithSource added in v0.1.1

func WithSource(job Job, source Source, ref string) Job

WithSource returns a copy of job whose labels carry source and ref (ref may be empty), leaving the caller's Labels map unchanged. It panics on nothing: an invalid source is caught by CheckJobLabels when the job is enqueued, and ValidateSource tells a caller earlier.

job = workflow.WithSource(job, "webhook", deliveryID)

type JournalOption

type JournalOption func(*journalPlugin)

JournalOption configures the Journal plugin.

func JournalKeepFailed

func JournalKeepFailed() JournalOption

JournalKeepFailed keeps the journal of a run that failed, so a later RunRetrier retry of that run resumes from its memoised work instead of redoing it. The app then forgets those runs itself (JournalStore.Forget, or the store's own retention). By default every finished run is forgotten, failed ones included, so an in-process journal never grows without bound.

type JournalStore

type JournalStore interface {
	// Load returns the value saved under key for run, and false when
	// there is none.
	Load(ctx context.Context, run RunID, key string) (val []byte, ok bool, err error)

	// Save stores val under key for run.
	Save(ctx context.Context, run RunID, key string, val []byte) error

	// Forget removes every entry of run. Forgetting a run with no
	// entries is not an error.
	Forget(ctx context.Context, run RunID) error
}

JournalStore keeps the results of memoised work inside a run (XM-02), so a step that is retried, or a run retried with RunRetrier (which keeps its RunRef), skips the work that already succeeded and gets its stored result back instead of doing it twice.

Entries are keyed by run and by a key Memo builds from the step name and the name the caller gave. A store must be safe for concurrent use, must return a copy (or a value it never changes again) from Load, and Save of an existing key replaces it. MemoryJournal is the in-process default; a durable store (SQL, Redis) makes the memo survive a crash of the process, which an in-process journal cannot.

type LifecycleEvent

type LifecycleEvent struct {
	Kind     LifecycleKind
	Ref      RunRef
	Workflow string
	Queue    string
	Tenant   string

	// Step and Attempt name the attempt for Started, Retrying, Snoozed
	// and Progress. For Finished, Step is the step whose failure ended the
	// run (RunEvent.Step) and Attempt is zero; for Inserted both are
	// empty.
	Step    string
	Attempt int

	// Status is the run's status after the event.
	Status Status

	// Err is the attempt's error (Retrying, Snoozed) or the run's
	// (Finished, failed); nil otherwise.
	Err error

	// Progress is set for LifecycleProgress.
	Progress *Progress

	// Node is the node the event happened on (CapNodeIdentity); empty
	// otherwise.
	Node string

	// Labels are the run's labels. Observers must not modify them.
	Labels map[string]string

	// At is when it happened, on the driver's clock.
	At time.Time
}

LifecycleEvent is one moment in a run's life, for a live view (a dashboard, a progress bar, an audit trail) that wants more than the end RunNotifier reports.

type LifecycleFunc

type LifecycleFunc func(ctx context.Context, ev LifecycleEvent)

LifecycleFunc observes lifecycle events.

type LifecycleKind

type LifecycleKind string

LifecycleKind is what happened to a run in a LifecycleEvent.

Code that switches on it must keep a default branch: a later version may add kinds.

const (
	// LifecycleInserted: Enqueue accepted a new run (not a duplicate
	// returned for an idempotency key).
	LifecycleInserted LifecycleKind = "inserted"
	// LifecycleStarted: a step attempt started (Step and Attempt set).
	LifecycleStarted LifecycleKind = "started"
	// LifecycleRetrying: a step attempt failed and the step will run
	// again after its backoff (Err set, Attempt is the failed one).
	LifecycleRetrying LifecycleKind = "retrying"
	// LifecycleSnoozed: a step snoozed (Snooze) and runs again later.
	LifecycleSnoozed LifecycleKind = "snoozed"
	// LifecycleProgress: a step reported progress (Progress set).
	LifecycleProgress LifecycleKind = "progress"
	// LifecycleFinished: the run reached a terminal status (Status set;
	// Err for a failure). The same moment RunNotifier reports.
	LifecycleFinished LifecycleKind = "finished"
)

The lifecycle kinds.

type LifecycleNotifier

type LifecycleNotifier interface {
	OnLifecycle(fn LifecycleFunc) error
}

LifecycleNotifier is an optional interface a client implements to stream its runs' lifecycle. The contract:

  • Register before Start: OnLifecycle after Start returns ErrAlreadyStarted. Each observer sees every event, in registration order.
  • Delivery is best-effort and local: an observer sees the events that happen in this client's process (an Enqueue it accepted, an attempt its workers ran), so on a cluster each node streams its own share. It is a live feed, not a log; RunNotifier with CapRunFinished is the guaranteed channel for run ends.
  • An observer runs on a driver goroutine, must return quickly, and a panic in it is recovered and never changes a run.
  • For one run, the events of one process arrive in the order they happened.

type LocalCron

type LocalCron struct {
	// contains filtered or unexported fields
}

LocalCron runs functions on a cron schedule inside this process, on every node that starts one: cache warmers, local temp cleanup, a per-node metrics flush. It is the opposite of Workflow.Cron and Schedule on a shared driver, which fire once across the fleet and may run the job on any worker; a LocalCron tick runs here, in-process, with no queue, no retries and no persistence.

Each entry runs on its own goroutine, one tick at a time: ticks that fall due while the previous run is still going are skipped, and ticks missed while the process was down are never caught up. An error or a panic goes to the error handler (WithLocalCronErrors) and the entry keeps its schedule.

Time comes from the injected clock (WithLocalCronClock, clock.Real by default), so a test drives it with clock.Fake.

func NewLocalCron

func NewLocalCron(opts ...LocalCronOption) *LocalCron

NewLocalCron returns a LocalCron with no entries.

func (*LocalCron) Add

func (l *LocalCron) Add(name, expr string, loc *time.Location, fn LocalCronFunc) error

Add registers fn to run at expr in loc. loc is required (time.UTC for UTC) and time.Local is refused, for the same reason as on Schedule. Names are unique. Every validation error wraps ErrInvalid; Add after Start returns ErrLocalCronStarted.

func (*LocalCron) Done

func (l *LocalCron) Done() <-chan struct{}

Done closes after Start's ctx ends and every entry has stopped.

func (*LocalCron) Start

func (l *LocalCron) Start(ctx context.Context) error

Start runs every entry until ctx ends, without blocking. Done closes once every entry's goroutine, including a run in progress, has returned.

type LocalCronFunc

type LocalCronFunc func(ctx context.Context, at time.Time) error

LocalCronFunc is the code a LocalCron entry runs. at is the tick it runs for (the logical time), not the wall time it started.

type LocalCronOption

type LocalCronOption func(*LocalCron)

LocalCronOption configures a LocalCron.

func WithLocalCronClock

func WithLocalCronClock(c clock.Clock) LocalCronOption

WithLocalCronClock sets the clock; nil keeps clock.Real.

func WithLocalCronErrors

func WithLocalCronErrors(f func(name string, at time.Time, err error)) LocalCronOption

WithLocalCronErrors receives every error and recovered panic of an entry's run, with the entry's name and tick. Nil (the default) drops them; the handler runs on the entry's goroutine and must not block.

func WithLocalCronParser

func WithLocalCronParser(p cron.Parser) LocalCronOption

WithLocalCronParser sets the cron dialect (cron.ParseSeconds for a seconds field); nil keeps cron.Parse.

type MapOption

type MapOption func(*mapConfig)

MapOption configures MapEach.

func MapConcurrency

func MapConcurrency(n int) MapOption

MapConcurrency runs at most n items at once. The default is 1 (one after another); n below 1 counts as 1.

type MemoryBuckets

type MemoryBuckets struct {
	// contains filtered or unexported fields
}

MemoryBuckets is an in-process BucketStore. The zero value is ready to use.

func (*MemoryBuckets) Observe

func (m *MemoryBuckets) Observe(_ context.Context, key string, spec BucketSpec, level float64, now time.Time) error

Observe implements BucketStore.

func (*MemoryBuckets) Take

func (m *MemoryBuckets) Take(_ context.Context, key string, spec BucketSpec, cost float64, now time.Time) (time.Duration, error)

Take implements BucketStore.

type MemoryJournal

type MemoryJournal struct {
	// contains filtered or unexported fields
}

MemoryJournal is an in-process JournalStore: entries live in this process only and are lost when it exits, so the memo holds across step retries and RunRetrier retries in one process, not across a crash. The zero value is ready to use.

func NewMemoryJournal

func NewMemoryJournal() *MemoryJournal

NewMemoryJournal returns an empty MemoryJournal.

func (*MemoryJournal) Forget

func (m *MemoryJournal) Forget(_ context.Context, run RunID) error

Forget removes every entry of run.

func (*MemoryJournal) Load

func (m *MemoryJournal) Load(_ context.Context, run RunID, key string) ([]byte, bool, error)

Load returns a copy of the value saved under key for run.

func (*MemoryJournal) Runs

func (m *MemoryJournal) Runs() []RunID

Runs returns the runs that hold at least one entry, sorted, for tests and for checking that finished runs were forgotten.

func (*MemoryJournal) Save

func (m *MemoryJournal) Save(_ context.Context, run RunID, key string, val []byte) error

Save stores a copy of val under key for run.

type Middleware

type Middleware func(next Handler) Handler

Middleware wraps every step execution, the same way on every driver: logging, tracing, recovery, timeouts and codecs are middleware, never driver wrappers that would hide what the driver can do.

func WithJournal

func WithJournal(store JournalStore) Middleware

WithJournal returns step middleware that puts store on every step's ctx, for Memo and the helpers built on it. A ctx without StepInfo (a handler called outside a driver) passes through unchanged, so Memo there returns ErrNoJournal. The Journal plugin bundles this with the cleanup of finished runs; use WithJournal alone only when something else forgets runs.

type Misfire

type Misfire struct {
	// Policy is what happens to missed ticks. Empty is the driver default.
	Policy MisfirePolicy

	// Within drops a missed tick reached more than this long after its
	// time, for MisfireFireOnce and MisfireFireAll (a starting deadline).
	// Zero means no limit. MisfireSkip uses MisfireGrace and refuses it.
	Within time.Duration

	// Max caps how many missed ticks MisfireFireAll fires per catch-up:
	// the newest Max, oldest first. Required (at least 1) for
	// MisfireFireAll and refused for the other policies, so a backfill is
	// never unbounded by accident.
	Max int
}

Misfire is a cron misfire policy. The zero value means the driver's default.

func (Misfire) Due

func (m Misfire) Due(src TickSource, last, now time.Time, loc *time.Location) []time.Time

Due returns the ticks of src in (last, now] that m fires, oldest first, in loc: the one rule every driver applies so the policies mean the same everywhere. last is when the scheduler last caught up (or started); now is the instant it woke.

  • MisfireSkip: the newest tick, only when now is within MisfireGrace of it; every older tick is dropped.
  • MisfireFireOnce: the newest tick, unless Within is set and it is more than Within old.
  • MisfireFireAll: every tick no more than Within old (any age when Within is zero), the newest Max of them.

The zero policy is the driver's own default, so Due returns nil for it and the driver applies its documented behaviour instead.

func (Misfire) IsZero

func (m Misfire) IsZero() bool

IsZero reports whether m is the driver default.

func (Misfire) Plan

func (m Misfire) Plan(src TickSource, last, now time.Time, loc *time.Location) (fire []time.Time, dropped int)

Plan is Due plus how many ticks the policy dropped, so a driver can report TickMisfired. The fire half is exactly Due's answer. The count covers ticks in (last, now] the policy did not fire, scanning in chunks so a long outage of a frequent cron costs bounded memory; it is capped at maxCountedTicks. The zero policy returns nil and 0, as Due does.

type MisfirePolicy

type MisfirePolicy string

MisfirePolicy is what a scheduler does with cron ticks it missed: ticks that fell due while nothing was running, or that it reached late.

const (
	// MisfireSkip drops every missed tick: a tick fires only when the
	// scheduler reaches it within MisfireGrace of its time (a 09:00 digest
	// must not go out at 14:00).
	MisfireSkip MisfirePolicy = "skip"
	// MisfireFireOnce fires the most recent missed tick once and drops the
	// older ones.
	MisfireFireOnce MisfirePolicy = "fire_once"
	// MisfireFireAll fires every missed tick, oldest first, at most
	// Misfire.Max of them per catch-up (a daily report per missed day).
	MisfireFireAll MisfirePolicy = "fire_all"
)

The misfire policies. A driver declares the ones it honours through CapMisfireSkip, CapMisfireFireOnce and CapMisfireFireAll and refuses the rest; the empty policy is the driver's own default and needs nothing.

type NextRun

type NextRun struct {
	// Local is the fire time in the schedule's zone.
	Local time.Time
	// UTC is the same instant in UTC.
	UTC time.Time
}

NextRun is one upcoming fire time shown two ways: on the schedule's wall clock and in UTC, so an operator in another zone reads both the time the schedule means and the instant it happens.

func NextRuns

func NextRuns(q NextRunsQuery) []NextRun

NextRuns lists the next q.Count fire times of q.Schedule that pass its Windows and Active period, each in the schedule's zone and in UTC (a dashboard's "next run" column). It returns fewer when the schedule ends or the scan limit is reached. It is a pure function: the reference time comes from q.After.

func (NextRun) String

func (r NextRun) String() string

String renders r as "2026-10-15 09:00:00 CEST (07:00:00 UTC)"; the date in parentheses is added when the UTC day differs.

type NextRunsQuery

type NextRunsQuery struct {
	// Schedule is the parsed expression (cron.Parse, or the driver's
	// parser).
	Schedule cron.Schedule
	// Location is the schedule's zone; nil means UTC.
	Location *time.Location
	// After is the instant to list from (exclusive): the caller's clock,
	// never read here.
	After time.Time
	// Count is how many fire times to list; at most MaxNextRuns.
	Count int
	// Windows and Active drop the fire times the schedule would skip, so
	// the list shows runs that will really start.
	Windows Windows
	Active  Active
}

NextRunsQuery describes the schedule whose upcoming fire times NextRuns lists.

type NodeInfo

type NodeInfo struct {
	// ID is the node's identity (StepInfo.Node).
	ID string

	// Mode is what the node does.
	Mode NodeMode

	// Queues are the queues its workers serve; empty means every queue
	// (or none for NodeScheduler and NodeEnqueueOnly).
	Queues []string

	// Workers is its worker pool size.
	Workers int

	// Running is how many step attempts it was running at LastSeen.
	Running int

	// StartedAt is when its client started; LastSeen when it last
	// reported in, on the driver's clock.
	StartedAt, LastSeen time.Time
}

NodeInfo is one node as NodeLister reports it.

type NodeLister

type NodeLister interface {
	Nodes(ctx context.Context) ([]NodeInfo, error)
}

NodeLister lists the nodes that are online, for an operator's view. It answers for itself: a client may implement it without CapNodeIdentity (an engine that lists its workers but keeps no per-attempt node), and callers check the interface, not the capability. Every client that declares CapNodeIdentity implements it. A node is online while it keeps reporting in; one that stops (a crash) drops out after the driver's expiry, which the driver documents.

type NodeMode

type NodeMode string

NodeMode is what a node does.

const (
	// NodeFull runs workers and the scheduler (the default).
	NodeFull NodeMode = "full"
	// NodeWorker runs workers only; another node fires schedules.
	NodeWorker NodeMode = "worker"
	// NodeScheduler fires schedules only, enqueueing the runs for worker
	// nodes (a separate scheduler process).
	NodeScheduler NodeMode = "scheduler"
	// NodeEnqueueOnly enqueues and answers queries, running nothing
	// (CapEnqueueOnly).
	NodeEnqueueOnly NodeMode = "enqueue-only"
)

The node modes.

type Op

type Op string

Op names an optional operation that carries no capability bit, so a caller can ask whether a client really does it before offering it (an admin button), instead of finding out from the call's refusal.

const (
	OpReschedule Op = "reschedule"  // RunRescheduler
	OpCancelStep Op = "cancel-step" // StepCanceller
	OpNodes      Op = "nodes"       // NodeLister
	OpLifecycle  Op = "lifecycle"   // LifecycleNotifier
)

The operations Supports answers for.

type OpProber

type OpProber interface {
	Supports(op Op) bool
}

OpProber is implemented by a client wrapper whose optional methods exist whether or not what it wraps can do them (Forward, failover): it answers for what is underneath.

type Outcome added in v0.1.1

type Outcome string

Outcome says whether a run's work succeeded, separately from its Status, which says whether the worker ran it to the end. A sync that skipped because nothing changed, or that finished with 3 bad rows out of 100, is StatusSucceeded either way; its outcome tells them apart. A step sets it with SetOutcome; a run recorder stores the last one set and derives one from the status when none was (DefaultOutcome).

Code that switches on it must keep a default branch: a later version may add outcomes.

const (
	// OutcomeSuccess: the work was done in full.
	OutcomeSuccess Outcome = "success"

	// OutcomePartial: some of the work was done, some was not (bad rows,
	// a page that failed); the run still succeeded.
	OutcomePartial Outcome = "partial"

	// OutcomeError: the work was not done.
	OutcomeError Outcome = "error"

	// OutcomeSkipped: there was nothing to do (no changes since the last
	// sync, a feature turned off).
	OutcomeSkipped Outcome = "skipped"
)

The outcomes.

func DefaultOutcome added in v0.1.1

func DefaultOutcome(status Status) Outcome

DefaultOutcome is the outcome of a run that ended with status and set none: OutcomeSuccess for StatusSucceeded, OutcomeError for StatusFailed and StatusDiscarded, and "" for every other status (a cancelled or unfinished run has no outcome yet).

func Outcomes added in v0.1.1

func Outcomes() []Outcome

Outcomes returns every outcome, as a fresh slice.

func (Outcome) Valid added in v0.1.1

func (o Outcome) Valid() bool

Valid reports whether o is one of the Outcome constants.

type OutcomeFunc added in v0.1.1

type OutcomeFunc func(ctx context.Context, o Outcome, note string) error

OutcomeFunc adapts a function to OutcomeSetter.

func (OutcomeFunc) SetOutcome added in v0.1.1

func (f OutcomeFunc) SetOutcome(ctx context.Context, o Outcome, note string) error

SetOutcome calls f.

type OutcomeSetter added in v0.1.1

type OutcomeSetter interface {
	SetOutcome(ctx context.Context, o Outcome, note string) error
}

OutcomeSetter stores the outcome a step sets. A run recorder binds one on the step's ctx (WithOutcomeSetter); drivers keep none.

type OverlapPolicy

type OverlapPolicy string

OverlapPolicy says what a schedule does with a tick that falls due while the run its previous fire started is still unfinished.

const (
	// OverlapAllow fires every tick whatever the previous run is doing.
	// It is the default (the zero value means it too).
	OverlapAllow OverlapPolicy = "allow"

	// OverlapSkip skips a tick while the schedule's previous run is not
	// terminal, and reports it as TickSkipped with TickReasonOverlap. The
	// decision reads the previous run's status from the driver's run
	// state, not a lock held by one node, so it holds across a fleet and
	// across a restart on a durable driver. A previous run the driver no
	// longer knows (deleted, pruned) does not block. It needs
	// CapOverlapSkip.
	OverlapSkip OverlapPolicy = "skip"
)

The overlap policies.

func (OverlapPolicy) Blocks

func (p OverlapPolicy) Blocks(prev Status) bool

Blocks reports whether p holds back a fire when the schedule's previous run has status prev. prev is empty when the schedule never fired or the driver no longer knows its previous run; that never blocks. Every driver decides overlap with this one function, so the rule cannot drift between backends.

type PauseTarget

type PauseTarget struct {
	// Workflow holds the runs of one workflow. Needs CapPauseWorkflow.
	Workflow string

	// Queue holds the runs on one queue (EffectiveQueue). Needs
	// CapPauseQueue.
	Queue string

	// All holds every run. Needs CapPauseWorkflow.
	All bool
}

PauseTarget names what Pauser holds: one workflow, one queue, or everything. Exactly one field is set; build it with PauseWorkflow, PauseQueue or PauseAll.

func PauseAll

func PauseAll() PauseTarget

PauseAll targets every run. Resume(PauseAll()) clears every pause.

func PauseQueue

func PauseQueue(name string) PauseTarget

PauseQueue targets the runs on one queue.

func PauseWorkflow

func PauseWorkflow(name string) PauseTarget

PauseWorkflow targets the runs of one workflow.

func (PauseTarget) Holds

func (t PauseTarget) Holds(wf, q string) bool

Holds reports whether a paused target t holds a run of workflow wf on queue q.

type Pauser

type Pauser interface {
	Pause(ctx context.Context, t PauseTarget) error
	Resume(ctx context.Context, t PauseTarget) error
	Paused(ctx context.Context) ([]PauseTarget, error)
}

Pauser holds runs without cancelling them, and resumes them. A client implements it when it declares CapPauseWorkflow or CapPauseQueue, and refuses the targets of the one it lacks (CheckPause).

While a target is paused, no step attempt of a run it holds is claimed: new runs and the next steps of running runs alike stay StatusQueued. Attempts already executing finish. Enqueue is still accepted. Timers (RunAt, sleeps, wait deadlines, retry backoffs) keep running, and what they make ready waits. A run is held while any paused target holds it. On a durable driver a pause is stored, so every node obeys it and it survives a restart.

Pause is idempotent. Resume removes exactly that target, except Resume(PauseAll()), which removes every pause; resuming a target that is not paused is a no-op. Paused lists the targets in force, All first, then workflows, then queues, each by name.

type PayloadLimiter

type PayloadLimiter interface {
	PayloadLimit() int
}

PayloadLimiter is an optional interface a driver's client implements to declare its hard payload cap in bytes (an engine's gRPC message limit, a broker's message size), so code above it (the claim check, a batch splitter) can size against the real number. Zero means no cap.

type PayloadTooLargeError

type PayloadTooLargeError struct {
	Workflow string
	Size     int
	Limit    int
	// Source says whose limit it is ("driver", "middleware", or a driver
	// name).
	Source string
}

PayloadTooLargeError says a payload is over a limit: the driver's hard cap (PayloadLimiter) or one the app set (middleware.MaxPayload). It matches ErrPayloadTooLarge and ErrInvalid, since the same bytes will never fit.

func (*PayloadTooLargeError) Error

func (e *PayloadTooLargeError) Error() string

Error names the sizes, so the message alone says by how much it is over.

func (*PayloadTooLargeError) Is

func (e *PayloadTooLargeError) Is(target error) bool

Is matches ErrPayloadTooLarge and ErrInvalid.

type Period

type Period string

Period is a calendar period for PeriodKey.

const (
	PeriodDay     Period = "day"
	PeriodWeek    Period = "week"
	PeriodMonth   Period = "month"
	PeriodQuarter Period = "quarter"
	PeriodYear    Period = "year"
)

The periods. Week is the ISO 8601 week (Monday first), so the last days of December can belong to week 1 of the next year.

type Plugin

type Plugin interface {
	Name() string
}

Plugin is the only interface a plugin must satisfy: a feature (unique jobs, batches, tracing, an audit log) bundled so an app adds it with one Use call instead of wiring each piece by hand.

Every contribution is an optional interface that Use type-asserts for: StepMiddlewareProvider, EnqueueMiddlewareProvider and RunFinishedProvider. A plugin implements only what it uses, so an absent contribution is absent rather than an empty method that looks the same as a forgotten one.

Invariant: Name is non-empty and unique among the plugins of one Use call. It names the plugin in every refusal, so an operator knows which one to remove or reconfigure.

func Journal

func Journal(store JournalStore, opts ...JournalOption) Plugin

Journal returns the plugin that turns on step memoisation: it puts store on every step's ctx (WithJournal) and forgets a run's entries when the run finishes. Pass it to Use before Register and Start; it needs a client with RunNotifier, as every run-finished plugin does.

type Progress

type Progress struct {
	// Percent is from 0 to 100.
	Percent float64

	// Message is a short human-readable note ("exported 4,000 of 10,000
	// orders").
	Message string

	// Data is optional app data, at most MaxProgressData bytes.
	Data []byte

	// Step is the step that reported it; the driver fills it.
	Step string

	// At is when it was reported, on the driver's clock; the driver
	// fills it.
	At time.Time
}

Progress is how far a run has got, as a step reported it from inside (ReportProgress): a percentage, a message for a person, and optional app data (rows exported so far, a cursor). The latest report wins.

func (Progress) Clone

func (p Progress) Clone() Progress

Clone returns a copy whose Data is its own.

type ProgressFunc

type ProgressFunc func(ctx context.Context, p Progress) error

ProgressFunc adapts a function to ProgressReporter.

func (ProgressFunc) ReportProgress

func (f ProgressFunc) ReportProgress(ctx context.Context, p Progress) error

ReportProgress calls f.

type ProgressReporter

type ProgressReporter interface {
	ReportProgress(ctx context.Context, p Progress) error
}

ProgressReporter stores a running step's progress with its run. A driver that keeps progress sets one on the step's ctx (WithProgressReporter); it fills Step and At, keeps the latest report as RunInfo.Progress, and (with a LifecycleNotifier) emits LifecycleProgress.

type QueueBounds

type QueueBounds struct {
	// Min is the fewest workers the queue keeps, however idle it is.
	// Zero means one.
	Min int
	// Max is the most workers it gets, however busy. Zero means the
	// whole budget (BalanceConfig.Total).
	Max int
}

QueueBounds limits one queue's worker count under a Balancer.

type QueueConfig

type QueueConfig struct {
	// Name is the queue; it must pass ValidateQueueName.
	Name string

	// Workers caps how many step attempts of this queue execute at once
	// on this client. Zero means no per-queue cap: only the driver's pool
	// size limits it.
	Workers int

	// Limit caps the in-flight step attempts of this queue across EVERY
	// client sharing the store, not only this one: Workers is per client
	// (per node), Limit is the queue's whole-cluster ceiling. Zero means no
	// cluster cap. It needs CapQueueLimitGlobal; a driver without it
	// refuses a non-zero Limit at Start (CheckQueueLimits). An attempt
	// holds its slot from claim to outcome, and a slot whose holder died is
	// freed when the attempt's lease expires, never leaked. Every client
	// that serves the queue should declare the same Limit: a client that
	// declares none does not enforce it (it still counts toward it).
	Limit int

	// Priority orders queues when a worker is free: the ready attempts of
	// a higher queue are always taken before a lower one's. Equal priorities fall back to Job.Priority,
	// then arrival. A driver that cannot order queues refuses a non-zero
	// Priority at Start.
	Priority int

	// Weight shares a worker between queues of equal Priority: when
	// several of them have ready work, a free worker picks one at random
	// in proportion to the weights (PickWeighted), so a queue of weight 3
	// gets about three times the turns of a queue of weight 1 and none
	// starves. Zero means strict order within the priority: Job.Priority,
	// then arrival, as before. A queue with weight among queues without
	// counts as weight 1. A driver that cannot weight queues refuses a
	// non-zero Weight at Start.
	Weight int
}

QueueConfig is how one client's workers serve one named queue. Drivers take a list of them through their own option (memory.WithQueues and its siblings); the type is shared so the meaning is the same everywhere.

A client given no QueueConfig serves every queue from one pool, as before queues existed. A client given some serves only those queues: on a shared store the runs of other queues wait for a client that serves them, which is how separate worker processes per queue are deployed.

type QueueLoad

type QueueLoad struct {
	// Queue is the queue name.
	Queue string

	// Depth is how many step attempts are ready and waiting for a worker
	// of this client (attempts in a retry backoff or a delayed start are
	// not counted: no worker could take them yet).
	Depth int

	// OldestWait is how long the oldest of those attempts has waited,
	// on the driver's clock; zero when Depth is zero.
	OldestWait time.Duration

	// Workers is the queue's current worker count on this client.
	Workers int

	// Running is how many of its attempts execute now.
	Running int
}

QueueLoad is one queue's load on one client, as a WorkerScaler reports it and a Balancer reads it.

type RateLimit

type RateLimit struct {
	// Key derives the limited bucket. Zero means one bucket for the
	// workflow.
	Key ConcurrencyKey

	// Limit is the number of starts allowed per window; at least 1.
	Limit int

	// Per is the window length; positive.
	Per time.Duration
}

RateLimit caps run starts per key value: at most Limit starts per window of length Per. The contract admits both fixed-window and sliding-window engines; for one key, every driver guarantees all three of:

  • (a) no span of length Per contains more than 2*Limit starts (a fixed window allows a burst at the end of one window and the start of the next, never more);
  • (b) N starts take at least (ceil(N/Limit) - 2) * Per from the first to the last (they fill at least ceil(N/Limit) windows, and a fixed window engine may start runs at the very end of the first and the very start of the last: Limit 2, six starts at 0:59.9, 1:00 and 2:00 span 60.1s, not 120s);
  • (c) within one window, at most Limit runs start.

A driver may be stricter (a sliding window keeps any span of length Per to Limit starts) and documents which kind it is. A run held back by the limit stays queued; it is delayed, never dropped.

type Recycle

type Recycle struct {
	// MaxJobs is how many step attempts this client may finish, whatever
	// their outcome. Attempts already running when it is reached still
	// finish, so a client with several workers may finish a few more.
	MaxJobs int

	// MaxTime is how long the client may run, from Start, on the
	// driver's clock.
	MaxTime time.Duration

	// MaxMemory is the memory reading, in bytes, above which the client
	// recycles. Checked every CheckEvery and after every attempt.
	MaxMemory uint64

	// Memory reads the current memory use in bytes. Nil means
	// HeapObjectBytes. A test passes a function returning what it needs;
	// an app may read RSS or a cgroup counter instead.
	Memory func() uint64

	// CheckEvery is how often the driver checks MaxMemory between
	// attempts. Zero means DefaultRecycleCheck.
	CheckEvery time.Duration
}

Recycle sets when a client stops itself so its process can be replaced: after MaxJobs step attempts, after MaxTime since Start, or once the memory reading passes MaxMemory. A zero field is no limit; the zero Recycle never recycles. When a limit is reached the client drains as Drain does (claims nothing more, lets running attempts finish), then stops; its StopCause is a *RecycleError. A driver takes it through its own option (memory.WithRecycle, sqlstore.WithRecycle).

func (Recycle) Enabled

func (r Recycle) Enabled() bool

Enabled reports whether any limit is set.

type RecycleError

type RecycleError struct {
	Reason RecycleReason
	// Jobs is how many step attempts the client had finished.
	Jobs int
	// Elapsed is the time from Start to the decision.
	Elapsed time.Duration
	// Memory is the last memory reading (zero when MaxMemory is unset).
	Memory uint64
}

RecycleError says which limit stopped a client. It matches ErrRecycle.

func (*RecycleError) Error

func (e *RecycleError) Error() string

func (*RecycleError) Is

func (e *RecycleError) Is(target error) bool

Is makes errors.Is(err, ErrRecycle) true.

type RecycleReason

type RecycleReason string

RecycleReason names the limit that stopped a client.

const (
	RecycleMaxJobs   RecycleReason = "max_jobs"
	RecycleMaxTime   RecycleReason = "max_time"
	RecycleMaxMemory RecycleReason = "max_memory"
)

The recycle reasons.

type RecycleWatch

type RecycleWatch struct {
	// contains filtered or unexported fields
}

RecycleWatch is the decision every driver shares: it counts finished attempts and checks the limits. It latches: once a limit is reached every later call returns the same error. Safe for concurrent use.

func NewRecycleWatch

func NewRecycleWatch(r Recycle, started time.Time) *RecycleWatch

NewRecycleWatch starts watching r from started (the driver's clock at Start).

func (*RecycleWatch) Check

func (w *RecycleWatch) Check(now time.Time) *RecycleError

Check checks every limit at now without counting an attempt.

func (*RecycleWatch) CheckEvery

func (w *RecycleWatch) CheckEvery() time.Duration

CheckEvery is the memory check interval: zero when MaxMemory is unset (no periodic check needed), else CheckEvery or DefaultRecycleCheck.

func (*RecycleWatch) Deadline

func (w *RecycleWatch) Deadline() (time.Time, bool)

Deadline is when MaxTime ends, and false when it is unset.

func (*RecycleWatch) JobDone

func (w *RecycleWatch) JobDone(now time.Time) *RecycleError

JobDone counts one finished attempt at now and checks every limit.

type Reporter

type Reporter interface {
	Report(ctx context.Context, f Failure)
}

Reporter receives failures for an error tracker (Sentry and the like). It must be safe for concurrent use and should not block for long: it is called on a driver goroutine. Errors are the reporter's to log.

type ReporterFunc

type ReporterFunc func(ctx context.Context, f Failure)

ReporterFunc adapts a function to Reporter.

func (ReporterFunc) Report

func (fn ReporterFunc) Report(ctx context.Context, f Failure)

Report calls fn.

type ReportingDrainer

type ReportingDrainer interface {
	Drainer
	DrainWithReport(ctx context.Context) (DrainReport, error)
}

ReportingDrainer is a Drainer that also says what it cut off. A client implements it when it declares CapDrainReport. DrainWithReport behaves exactly as Drain (same shutdown, same errors) and additionally returns the report. Called again, or after the client stopped, it returns the same report as the first call: the report describes one drain.

type ResultReader

type ResultReader interface {
	Result(ctx context.Context, ref RunRef) ([]byte, error)
}

ResultReader returns a succeeded run's result. A client implements it when it declares CapResult.

The result is RunOutput of the run's steps: kept with the run until the run is deleted or pruned by retention. A run that is not StatusSucceeded returns ErrWrongState (ErrNotFound when unknown), so a caller can tell "not done" from "done with an empty result".

type RetryPolicy

type RetryPolicy struct {
	// Max is the number of retries after the first attempt, so a step runs
	// at most Max+1 times. The value that applies is the one of the policy
	// EffectiveRetriesFor resolves: job over step over workflow over
	// DefaultRetryMax.
	Max int

	// Backoff is the wait between attempts. Zero means retry at once.
	Backoff Backoff

	// NoRetryOn marks errors that must not be retried. Returning
	// Permanent(err) from the handler does the same without a policy.
	// It is a function, so it is never encoded: JSON skips it.
	NoRetryOn func(error) bool `json:"-"`

	// DelayFor sets the wait before a retry from the error itself, such
	// as a 429's Retry-After: it returns the delay and true, or false to
	// fall back to Backoff. retry is 1 for the first retry. The delay is
	// used as returned (no jitter; a negative one is zero). Nil means
	// Backoff alone. RetryAfterDelay is the stock implementation, reading
	// errors wrapped by RetryAfter. Needs CapRetryDelay.
	// It is a function, so it is never encoded: JSON skips it.
	DelayFor func(err error, retry int) (time.Duration, bool) `json:"-"`

	// Until stops retrying once a retry would start more than Until after
	// the step's first attempt started. It wins
	// over Max: a step with Max 100 and Until one hour stops at the hour.
	// Zero means no deadline. Needs CapRetryUntil.
	Until time.Duration

	// MaxErrors fails the step for good on its MaxErrors-th error, even
	// when Max allows more attempts. A release
	// (an error marked by RetryAfter, see IsRelease) counts against Max
	// only; any other error counts against both. Zero means no cap. Needs
	// CapRetryMaxErrors.
	MaxErrors int

	// DiscardOnExhaust ends the run StatusDiscarded (Err matching
	// ErrDiscarded and ErrCancelled, no OnFailure, no chain) instead of
	// StatusFailed when this policy stops retrying a failed attempt:
	// attempts ran out (Max), MaxErrors or Until stopped it, NoRetryOn
	// excluded the error, or the handler returned Permanent. Use it for
	// work whose giving up is routine (a notification nobody needs after
	// five tries) so failed stays a signal that needs a person. The default
	// (false) keeps the run failed. An explicit Discard and ErrCancelled are
	// not affected. Needs CapDiscardExhausted, which also needs
	// CapRunStates to report the status.
	DiscardOnExhaust bool
}

RetryPolicy says how often and how soon a failed step is tried again. Its zero value means no retries (Max is DefaultRetryMax).

A policy reaches a step from four places; the first that is set wins and the others are ignored whole (policies are never merged field by field):

  1. Job.Retry, chosen per dispatch, for every step of that run;
  2. Step.Retry (RetriesNone() means never retry);
  3. Workflow.Retry, the default for steps whose Retry is nil;
  4. the zero policy, DefaultRetryMax retries.

EffectiveRetriesFor is the one function that applies this order, and every driver calls it, so no two drivers can resolve a step's policy differently.

func EffectiveRetries

func EffectiveRetries(step Step, wf Workflow) RetryPolicy

EffectiveRetries resolves the policy a driver must apply to step inside wf when no per-dispatch override exists: the step's own Retry when set, else the workflow default. It is EffectiveRetriesFor with an empty Job, for callers that decide at declaration time (a driver that registers retries with an engine); a driver that sees runs calls EffectiveRetriesFor.

func EffectiveRetriesFor

func EffectiveRetriesFor(step Step, wf Workflow, job Job) RetryPolicy

EffectiveRetriesFor is the one place the retry precedence lives, job over step over workflow over default: job.Retry when set (a per-dispatch override wins over every declared policy), else step.Retry, else wf.Retry, whose zero value is the default of DefaultRetryMax retries. The winning policy is returned whole; nothing is merged. Every driver that retries (memory, sqlstore, redis, RunInline) calls it with the run's own Job, so a driver that ignored Job.Retry on a step would be a bug the conformance suite catches.

func RetriesNone

func RetriesNone() *RetryPolicy

RetriesNone returns a policy that never retries, for a step that must not inherit the workflow default. It returns a fresh value each call so no caller can change it for another.

func (RetryPolicy) ExhaustedError

func (p RetryPolicy) ExhaustedError(attempts int, err error) error

ExhaustedError is what a driver ends a run with when policy stopped retrying a step's failed attempt number attempts with err and policy.DiscardOnExhaust is set: the returned error is a Discard of err (IsDiscarded), so EndStatus maps it to StatusDiscarded and DiscardedError builds the run's Err from it. With the option off, or when err already is a discard or a cancellation (those end the run their own way), it returns nil and the driver fails the run as before. Every driver calls it at the point Next says stop, so none decides differently.

func (RetryPolicy) Next

func (p RetryPolicy) Next(err error, a Attempts, now time.Time, rnd func() float64) (delay time.Duration, retry bool)

Next is the one retry decision every driver makes when a step attempt fails with err: whether to try again and, if so, after how long. It applies, in order:

  • Retryable(err, a.N): nil, Permanent, Discard, ErrCancelled and NoRetryOn stop it, and so does a.N beyond Max;
  • MaxErrors: a.Errors reaching it stops it;
  • the delay, NextDelay(err, a.N, rnd);
  • Until: a retry starting after a.First + Until (now + delay) stops it.

now is the driver's clock. It is a pure function of its arguments (with a pinned rnd), so a test can pin every branch.

func (RetryPolicy) NextDelay

func (p RetryPolicy) NextDelay(err error, retry int, rnd func() float64) time.Duration

NextDelay is the wait before retry number retry (1 for the first) after err: DelayFor's answer when it gives one, else the Backoff delay with its jitter drawn from rnd, a source of uniform values in [0, 1) (nil means math/rand/v2). Every driver that retries calls it, so no two drivers can disagree on the rule.

func (RetryPolicy) Retryable

func (p RetryPolicy) Retryable(err error, attempts int) bool

Retryable reports whether err, after the given number of attempts (1 for the first), may be tried again: not permanent, not discarded, not excluded by NoRetryOn, and within Max. It ignores Until and MaxErrors, which need more than an attempt count; drivers call Next.

type RunDeleter

type RunDeleter interface {
	Delete(ctx context.Context, ref RunRef) error
}

RunDeleter forgets a terminal run: afterwards Status returns ErrNotFound and Runs no longer lists it. A client implements it when it declares CapDeleteRun. A run that is not terminal returns ErrWrongState; an unknown ref returns ErrNotFound. Forgetting many runs is Runs plus Delete, or a driver's retention.

type RunError

type RunError struct {
	RunID    RunID
	Workflow string
	Step     string
	Attempt  int
	Err      error
}

RunError is the error a driver records for a run that failed: it carries the run, the workflow, the step and the attempt as fields, so code holding the error (a RunEvent.Err, an error tracker's hook) finds which run to look up with RunErrorOf or errors.As. Its Error text is the step's own message unchanged, so RunInfo.Error, logs and dashboards keep showing what the app said; Describe gives the message with the context in front. It wraps the step's own error, so errors.Is and errors.As keep working through it.

func RunErrorOf

func RunErrorOf(err error) (*RunError, bool)

RunErrorOf returns the RunError err carries, if any.

func (*RunError) Describe

func (e *RunError) Describe() string

Describe reads `workflow "send-email" run r-12 step "deliver" attempt 3: <err>`, for a log line that has nothing else to say which run it was.

func (*RunError) Error

func (e *RunError) Error() string

Error is the step's own message (Err.Error()).

func (*RunError) Unwrap

func (e *RunError) Unwrap() error

Unwrap returns the step's error.

type RunEvent

type RunEvent struct {
	// Ref is the run. ScheduledID is set too when the run was enqueued
	// with RunAt, so a caller that stored the RunRef Enqueue returned
	// (which held only the ScheduledID) can match the event.
	Ref RunRef

	// Workflow is the workflow name.
	Workflow string

	// Status is terminal: StatusSucceeded, StatusFailed or
	// StatusCancelled, or StatusDiscarded on a driver with CapRunStates.
	Status Status

	// Err is why the run failed; nil when it succeeded. In-process
	// drivers pass the original error; an engine rebuilds it from the
	// message it stored, so errors.Is against an app's own sentinel is
	// reliable only on in-process drivers. A cancelled run's Err matches
	// ErrCancelled on every driver.
	Err error

	// Labels are the run's Job.Labels. Observers must not modify them.
	Labels map[string]string

	// FinishedAt is when the run reached its terminal status, from the
	// driver's clock: the engine's timestamp for an engine driver when the
	// engine records one, else the driver's own clock at the moment it
	// learned of the end (a delayed run cancelled before it fired).
	FinishedAt time.Time

	// Step is the step whose failure ended the run (RunInfo.FailedStep);
	// empty for other ends and for drivers that do not know it.
	Step string

	// Tenant is the run's Job.Tenant (CapTenant).
	Tenant string
}

RunEvent reports that a run reached a terminal status. It is what lets app-side code react to a run's end without polling Status: an enqueue middleware releasing a uniqueness key, a batch counting finished members, an audit log.

type RunFailedError

type RunFailedError struct {
	Ref    RunRef
	Status Status
	// Err is RunInfo.Error when the client could say why; empty otherwise.
	Err string
}

RunFailedError is what AwaitResult returns for a run that ended without succeeding. It matches ErrCancelled for a cancelled or discarded run.

func (*RunFailedError) Error

func (e *RunFailedError) Error() string

Error names the run and why it ended.

func (*RunFailedError) Is

func (e *RunFailedError) Is(target error) bool

Is makes a cancelled or discarded run match ErrCancelled.

type RunFinishedFunc

type RunFinishedFunc func(ctx context.Context, ev RunEvent)

RunFinishedFunc observes run ends. It must be idempotent, because delivery is at least once.

type RunFinishedProvider

type RunFinishedProvider interface {
	RunFinishedObservers() []RunFinishedFunc
}

RunFinishedProvider contributes run-finished observers. Use registers them on the client's RunNotifier, so it must run before Start, and it refuses a client without RunNotifier rather than dropping them. The delivery contract, best-effort included, is RunNotifier's.

type RunForcer

type RunForcer interface {
	ForceComplete(ctx context.Context, ref RunRef) error
	ForceFail(ctx context.Context, ref RunRef, reason string) error
}

RunForcer ends a run from outside, for an operator handling a work item a step cannot finish on its own. A client implements it when it declares CapForceRun.

ForceComplete ends a run as StatusSucceeded without running its remaining steps. It accepts a run that is not terminal, or one that is StatusFailed (accepting a failure as done). Attempts in progress have their context cancelled with cause ErrForced and their results are ignored. The run is then treated exactly as a success: its idempotency key is released, its Job.Chain continues, and run-finished observers see StatusSucceeded (again, for a run that had failed: they already tolerate repeats). Other terminal statuses return ErrWrongState.

ForceFail ends a run that is not terminal as StatusFailed, with an error matching ErrForced that carries reason. No retry is made and OnFailure does not run (the operator has decided; it is not the workflow's failure path); RunRetrier can still re-run it. A terminal run returns ErrWrongState.

Both return ErrNotFound for an unknown ref. Neither moves a run to another worker or queue: re-running a failed run elsewhere is RunRetrier (any worker may pick it up).

type RunID

type RunID string

RunID identifies one run. Drivers treat it as opaque: the memory driver uses a counter, an engine uses its own ID format.

const InlineRunID RunID = "inline"

InlineRunID is the RunID a step sees in its StepInfo under RunInline.

func (RunID) IsZero

func (id RunID) IsZero() bool

IsZero reports whether the ID is unset.

func (RunID) String

func (id RunID) String() string

String returns the ID as a string.

type RunInfo

type RunInfo struct {
	// Ref is the run.
	Ref RunRef

	// Workflow is the workflow name.
	Workflow string

	// Queue is the run's queue (EffectiveQueue).
	Queue string

	// Status is where the run is now.
	Status Status

	// Error is why the run failed, was discarded or was cancelled; while
	// it is StatusRetrying (CapRunStates), the error of the failed attempt
	// that waits on its backoff; empty otherwise. It is a string because
	// an engine stores only the message.
	Error string

	// FailedStep is the step whose failure ended the run; empty otherwise.
	FailedStep string

	// Labels are the run's Job.Labels. Callers must not modify them.
	Labels map[string]string

	// Payload is the job's payload as the driver stored it (after any
	// codec or claim check ran).
	Payload []byte

	// Priority is Job.Priority.
	Priority int

	// EnqueuedAt is when Enqueue accepted the run, on the driver's clock.
	EnqueuedAt time.Time

	// StartedAt is when the first step attempt started; zero before.
	StartedAt time.Time

	// FinishedAt is when the run became terminal; zero before.
	FinishedAt time.Time

	// Attempts counts the step attempts made across the run.
	Attempts int

	// NextAttempt is the attempt number a step waiting on its retry
	// backoff will run as (2 for the first retry), and NextAttemptAt
	// when it becomes ready; with several steps in backoff, the one
	// ready first. Both are zero unless the run is StatusRetrying, and
	// are filled only by drivers with CapRunStates.
	NextAttempt   int
	NextAttemptAt time.Time

	// Errors is the run's error history: one entry per failed step
	// attempt, oldest first, bounded by the driver's error-history
	// option. Filled only by drivers with CapErrorHistory.
	Errors []AttemptError

	// Tenant is Job.Tenant (CapTenant).
	Tenant string

	// RunAt is the time a delayed run is (or was) due: Job.RunAt, or the
	// time Reschedule moved it to. Zero for a run enqueued to start now.
	// Filled by drivers with CapDueQuery.
	RunAt time.Time

	// Output is the run's result (RunOutput) once it succeeded; nil
	// before and for other ends. Filled by drivers with CapResult.
	Output []byte

	// Progress is the latest progress a step reported (ReportProgress);
	// nil when none was reported or the driver keeps none.
	Progress *Progress

	// AttemptedBy lists the nodes that ran the run's step attempts, in
	// the order each first ran one, without repeats. Filled by drivers
	// with CapNodeIdentity.
	AttemptedBy []string
}

RunInfo is one run as a RunQuerier returns it.

func (RunInfo) Clone

func (info RunInfo) Clone() RunInfo

Clone returns a copy of info whose Labels and Payload are its own, so a driver can hand it out without sharing its state.

type RunNotifier

type RunNotifier interface {
	// OnRunFinished registers fn. It returns ErrAlreadyStarted after
	// Start.
	OnRunFinished(fn RunFinishedFunc) error
}

RunNotifier is an optional interface a driver's client implements to report run ends. Callers discover it with a type assertion on the Client, so a driver that cannot report run ends is never forced to pretend it can, and app-side code falls back (for example to a TTL) when the assertion fails.

The delivery contract every implementation keeps:

  • Register before Start: OnRunFinished after Start returns ErrAlreadyStarted. Several observers may be registered; each sees every event, in registration order.
  • An observer runs on a driver goroutine, never on the goroutine that enqueued or cancelled the run, and may run concurrently for different runs. It should return quickly; long work belongs in a new Enqueue.
  • It is called after Status reports the terminal status, so an observer that calls Status sees the same status as ev.Status.
  • Its ctx is cancelled when the client shuts down (the Start ctx).
  • A panic in an observer is recovered by the driver and never changes the run's status.
  • Delivery is at least once: an observer may see the same run's event more than once (an engine retrying a delivery) and must tolerate it.
  • With CapRunFinished, every run that reaches a terminal status is delivered, including cancelled runs and runs that never started; a durable driver also delivers, after a restart, the ends that happened while no client was running. Without CapRunFinished, delivery is best-effort: an event may be lost (for example on a crash), so a consumer must keep a fallback.

type RunPage

type RunPage struct {
	Runs []RunInfo
	Next string
}

RunPage is one page of a run listing, newest enqueue first (ties broken by RunID, descending). Next is empty on the last page.

type RunQuerier

type RunQuerier interface {
	Runs(ctx context.Context, q RunQuery) (RunPage, error)
}

RunQuerier lists runs, failed ones included, for an operator's view. A client implements it when it declares CapRunQuery. Errors: an invalid query wraps ErrInvalid; a cursor the driver did not issue wraps ErrInvalid too.

type RunQuery

type RunQuery struct {
	// Ref picks one run: every field of Ref that is set must equal the
	// run's, so a RunRef that Enqueue returned for a delayed run (only its
	// ScheduledID) still finds the run after it started. The zero RunRef
	// matches any run. With Ref set, Runs returns zero or
	// one run, which is how an admin surface shows a single run.
	Ref RunRef

	// Workflow is the workflow name; empty matches any.
	Workflow string

	// Queue is the run's queue (EffectiveQueue); empty matches any.
	Queue string

	// Statuses matches runs in any of these statuses; empty matches any
	// (CancelAll gives it its own default).
	Statuses []Status

	// Labels matches runs whose Job.Labels hold every pair exactly.
	Labels map[string]string

	// LabelPrefixes matches runs whose Job.Labels hold every key with a
	// value starting with the given prefix (an empty prefix matches any
	// value under that key). It needs CapLabelPrefix; a driver without it
	// refuses the query (CheckRunQuery) rather than ignoring the filter.
	LabelPrefixes map[string]string

	// From and To bound the enqueue time to [From, To); a zero bound is
	// open.
	From, To time.Time

	// Tenant matches runs whose Job.Tenant equals it; empty matches any.
	// It needs CapTenant; a driver without it refuses the query
	// (CheckRunQuery).
	Tenant string

	// DueFrom and DueTo bound the time a delayed run is due (RunInfo.RunAt)
	// to [DueFrom, DueTo), for "what fires in the next hour". With either
	// set, only runs enqueued with a RunAt match. A zero bound is open.
	// They need CapDueQuery (CheckRunQuery).
	DueFrom, DueTo time.Time

	// Limit is the page size: zero means DefaultRunsLimit, more than
	// MaxRunsLimit is invalid. CancelAll refuses it.
	Limit int

	// Cursor continues a listing from RunPage.Next; empty starts at the
	// newest run. It is opaque and driver-specific. CancelAll refuses it.
	Cursor string
}

RunQuery filters runs for RunQuerier.Runs and BulkCanceller.CancelAll. Every field that is set must match; the zero value matches every run.

func (RunQuery) Match

func (q RunQuery) Match(info RunInfo) bool

Match reports whether a run with these fields passes q's filters (everything but paging). In-process drivers use it so every driver reads a filter the same way.

func (RunQuery) PageLimit

func (q RunQuery) PageLimit() int

PageLimit is the page size q asks for, after the default.

type RunRef

type RunRef struct {
	RunID       RunID
	ScheduledID ScheduledID
}

RunRef points at a run. ScheduledID is set while a delayed run has not started; RunID is set once it has. A driver that learns the RunID of a delayed run resolves it itself, so callers keep passing the RunRef they were given.

func EnqueueClosure

func EnqueueClosure(ctx context.Context, client Client, c ClosureJob) (RunRef, error)

EnqueueClosure runs c on client when it implements ClosureEnqueuer and declares CapClosure, and otherwise returns an UnsupportedError naming CapClosure, never silently dropping the work.

func Spawn

func Spawn(ctx context.Context, client Client, name string, jobs []Job, opts ...SpawnOption) ([]RunRef, error)

Spawn enqueues jobs from inside a step as runs of their own and returns their RunRefs in order, without waiting for them: multiple instances without synchronization (WC-12). Each child is its own run, with its own retries and its own worker, and carries LabelParentRun and LabelParentStep.

A retried step does not spawn twice: each job's RunRef is memoised under name and the job's index (Spawn needs the Journal plugin, as Memo does), and a job without an IdempotencyKey gets one derived from the run, the step, name and the index, which covers a crash between the enqueue and the save. That key needs CapIdempotencyKey on client; a client without it is refused with an UnsupportedError unless SpawnAllowDuplicates is given. A job that already has a key keeps it.

client is the client the app enqueues with (a step has no client of its own); pass the same one the step's workflow is registered on, or another one to spawn onto another backend.

func (RunRef) IsZero

func (r RunRef) IsZero() bool

IsZero reports whether the ref points at nothing.

type RunRescheduler

type RunRescheduler interface {
	Reschedule(ctx context.Context, ref RunRef, at time.Time) (RunRef, error)
}

RunRescheduler moves a delayed run that has not started to a new time.

Reschedule needs a run in StatusScheduled (a Job.RunAt enqueue not yet due): it runs at `at` instead, keeps its RunRef, payload, labels and idempotency key, and RunInfo.RunAt reads `at`. An `at` in the past (or zero) means now. Any other status returns ErrWrongState, an unknown ref ErrNotFound. A client without it returns an UnsupportedError from the Forward wrapper; the fallback is Cancel plus Enqueue.

type RunRetrier

type RunRetrier interface {
	Retry(ctx context.Context, ref RunRef) (RunRef, error)
}

RunRetrier re-runs a failed run from the step that failed, without re-running the steps that already succeeded. A client implements it when it declares CapRetryRun.

Steps that succeeded keep their outputs and do not run again; the step that failed, and every step the failure cancelled, start again with fresh attempt counts. The run keeps its RunRef, goes back to StatusQueued and, when it ends again, is reported again to run-finished observers (which already tolerate repeats). Its idempotency key is taken again; when another unfinished run holds it, Retry returns ErrDuplicate. A run that is not StatusFailed returns ErrWrongState; an unknown ref returns ErrNotFound.

type Schedule

type Schedule struct {
	// Name is a stable, human-readable name an engine may show. A
	// schedule with a Name is upserted by (Workflow, Name) on every
	// driver: calling Schedule again with the same name returns the same
	// ScheduleID and replaces that schedule instead of adding a second, so
	// an app can declare its schedules at every boot. Without a Name every
	// call adds a schedule.
	Name string

	// Workflow is the registered workflow to run.
	Workflow string

	// Cron is a 5-field cron expression.
	Cron string

	// Location is the time zone Cron is evaluated in. It is required, so
	// no schedule is ever evaluated in an implicit zone; use time.UTC for
	// UTC. time.Local is refused for the same reason: it differs from
	// machine to machine. A driver whose engine only runs UTC crons returns an
	// UnsupportedError for any other zone rather than shifting it.
	Location *time.Location

	// Input is the payload of each fired run.
	Input []byte

	// Labels are copied onto each fired run's Job.Labels. Keys under a
	// reserved prefix (ReservedLabelPrefixes) are refused.
	Labels map[string]string

	// Misfire says what happens to ticks the driver reached late or not
	// at all. The zero value is the driver's default.
	Misfire Misfire

	// Windows limits ticks to time-of-day windows on Location's wall
	// clock (only between, or never between, two times). A tick outside them is
	// skipped, which tick observers see as TickSkipped with Reason
	// TickReasonWindow. Zero means no limit. Needs CapScheduleWindow.
	Windows Windows

	// Overlap is the schedule's overlap policy: OverlapSkip skips a tick
	// (and refuses a manual trigger with ErrOverlap) while the run of
	// its previous fire is unfinished. The zero value is OverlapAllow.
	Overlap OverlapPolicy

	// Seconds says Cron has six fields, seconds first (cron.ParseSeconds).
	// Needs CapCronSeconds.
	Seconds bool

	// Active limits the schedule to a period (see Active). Zero means
	// always. Needs CapScheduleActive.
	Active Active

	// KeepRuns keeps only the newest N finished runs this schedule fired;
	// older finished runs are forgotten as RunDeleter would. Zero keeps
	// them all. Needs CapScheduleKeep.
	KeepRuns int

	// Tenant is the Job.Tenant of every run the schedule fires, by tick or
	// by TriggerSchedule, so a per-tenant schedule's runs are found by
	// RunQuery.Tenant like any other run of that tenant. Empty means none.
	// It must pass ValidateTenant and is stored with the schedule. Needs
	// CapTenant.
	Tenant string
}

Schedule is a recurring trigger made at run time (per tenant, per user), as opposed to Workflow.Cron, which is declared with the workflow.

func UnmarshalSchedule

func UnmarshalSchedule(data []byte, loadZone func(name string) (*time.Location, error)) (Schedule, error)

UnmarshalSchedule reads a Schedule's JSON form. loadZone resolves the zone name; nil means cron.LoadZone, which refuses "Local" and unknown zones, so a stored schedule with a mistyped zone fails at load, not as a schedule firing at the wrong hour. A version other than ScheduleJSONVersion and unknown fields are refused. Every error wraps ErrScheduleJSON.

func (Schedule) MarshalJSON

func (s Schedule) MarshalJSON() ([]byte, error)

MarshalJSON writes s in its versioned JSON form (ScheduleJSONVersion), for storing schedules as data: a settings table, a config file, an admin form. The zone is written by its IANA name, so a Location without one (time.Local, a time.FixedZone) is refused rather than written as a name no reader can load. It does not validate the rest; Validate does.

func (*Schedule) UnmarshalJSON

func (s *Schedule) UnmarshalJSON(data []byte) error

UnmarshalJSON reads the form MarshalJSON writes, loading the zone with cron.LoadZone. Use UnmarshalSchedule to load zones another way.

type ScheduleHooks

type ScheduleHooks struct {
	// Workflow is the schedule's workflow; required.
	Workflow string

	// Name is the Schedule.Name (for a declared cron, its expression);
	// empty matches every schedule of Workflow.
	Name string

	// Fired is called when a tick enqueued a run, or failed to (t.Err).
	Fired func(ctx context.Context, t Tick)

	// Skipped is called when a tick was skipped or misfired.
	Skipped func(ctx context.Context, t Tick)

	// OnSuccess is called when a run the schedule fired succeeded.
	OnSuccess func(ctx context.Context, ev RunEvent)

	// OnFailure is called when a run the schedule fired failed.
	OnFailure func(ctx context.Context, ev RunEvent)

	// After is called when a run the schedule fired ended, in any status.
	After func(ctx context.Context, ev RunEvent)
}

ScheduleHooks are before, on-success, on-failure and after callbacks for one schedule, or for every schedule of a workflow, built only from TickNotifier and RunNotifier so they work on any driver that has both. Ping a heartbeat service from After or OnSuccess.

Every hook is optional. They run on the driver's observer goroutines with that contract: at least once, so a hook must be idempotent, and best-effort unless the driver declares CapRunFinished and CapTickEvents.

type ScheduleID

type ScheduleID string

ScheduleID identifies a recurring trigger made by Client.Schedule. The app stores it and passes it back to Unschedule.

func (ScheduleID) IsZero

func (id ScheduleID) IsZero() bool

IsZero reports whether the ID is unset.

func (ScheduleID) String

func (id ScheduleID) String() string

String returns the ID as a string.

type ScheduleInfo

type ScheduleInfo struct {
	// ID is the schedule. A declared cron has the driver's stable id for
	// that declaration.
	ID ScheduleID

	// Name is the Schedule.Name; for a declared cron, the expression.
	Name string

	// Workflow is the workflow the schedule runs.
	Workflow string

	// Cron is the expression.
	Cron string

	// Location is the zone it is evaluated in.
	Location *time.Location

	// Declared is true for a Workflow.Cron entry.
	Declared bool

	// Paused is true while SchedulePauser holds it.
	Paused bool

	// Next is the next fire time in Location; zero when the expression
	// never fires again. It ignores windows and ScheduleWhen, which are
	// decided at tick time.
	Next time.Time

	// Misfire is the schedule's misfire policy.
	Misfire Misfire

	// Windows are the schedule's time-of-day windows.
	Windows Windows

	// Labels are the schedule's labels. Callers must not modify them.
	Labels map[string]string

	// Overlap is the schedule's overlap policy (for a declared cron,
	// Workflow.CronOverlap); empty means OverlapAllow.
	Overlap OverlapPolicy

	// Last is the schedule's most recent fire, filled by drivers with
	// CapScheduleHistory; zero otherwise.
	Last ScheduleLast

	// Seconds is Schedule.Seconds (Workflow.CronSeconds for a declared
	// cron).
	Seconds bool

	// Active is the schedule's period (Workflow.CronActive for a declared
	// cron). Next already respects it.
	Active Active

	// KeepRuns is Schedule.KeepRuns (Workflow.CronKeepRuns).
	KeepRuns int

	// Tenant is Schedule.Tenant; empty for a declared cron.
	Tenant string
}

ScheduleInfo is one live schedule as ScheduleLister reports it.

type ScheduleLast

type ScheduleLast struct {
	// Fire is when the schedule last enqueued a run: the tick's time for
	// a cron fire, the driver's clock for a trigger. Zero when it never
	// fired.
	Fire time.Time

	// Run is the run that fire enqueued. Zero when it never fired.
	Run RunRef

	// Status is that run's status when the schedule was listed; empty
	// when it never fired or the driver no longer knows the run (deleted,
	// pruned). It is the "last outcome" once the run is terminal.
	Status Status
}

ScheduleLast is the bookkeeping of a schedule's most recent fire, cron tick or manual trigger alike. Drivers with CapScheduleHistory store it with the schedule (durably on a durable driver) and report it in ScheduleInfo.

type ScheduleLister

type ScheduleLister interface {
	Schedules(ctx context.Context) ([]ScheduleInfo, error)
}

ScheduleLister lists the live schedules with their next fire time, declared and made at run time, sorted by Workflow, then Name, then ID. Unscheduled ones are not listed. A client implements it when it declares CapScheduleList.

type SchedulePauser

type SchedulePauser interface {
	PauseSchedule(ctx context.Context, t ScheduleTarget) error
	ResumeSchedule(ctx context.Context, t ScheduleTarget) error
}

SchedulePauser stops schedules firing without removing them, and resumes them. A client implements it when it declares CapSchedulePause.

A paused schedule's ticks do not fire and are reported as TickSkipped with TickReasonPaused. A tick that fell due during the pause never fires, not even on resume and whatever the misfire policy (a pause means "not now", not "later"): the schedule resumes at its next tick, so each tick is reported once. The misfire policy still governs ticks missed while nothing was running. Pausing All pauses every schedule, including ones made after the call; ResumeSchedule with All clears every pause. Pause and resume are idempotent; an unknown ID returns ErrNotFound. On a durable driver the pause is stored.

type ScheduleTarget

type ScheduleTarget struct {
	ID  ScheduleID
	All bool
}

ScheduleTarget names a schedule for SchedulePauser, or all of them. Exactly one field is set.

type ScheduleTriggerer

type ScheduleTriggerer interface {
	TriggerSchedule(ctx context.Context, id ScheduleID) (RunRef, error)
}

ScheduleTriggerer runs a live schedule now, outside its cron, with the schedule's Input and labels (FiredLabels plus LabelTrigger), as if a tick had fired at the driver's current time. A client implements it when it declares CapScheduleTrigger.

A trigger obeys the schedule's OverlapPolicy: with OverlapSkip and the previous run unfinished it returns ErrOverlap and enqueues nothing. It does not consult Windows or ScheduleWhen (an operator asking for a run now has already decided), and it fires a paused schedule too (a pause stops the cron, not the operator). The triggered run becomes the schedule's last run (ScheduleInfo.Last: Fire and Run), so a later tick's overlap check sees it, and drivers with CapTickEvents report it as a TickFired with Tick.Manual set. The schedule's next cron fire is unchanged. An unknown or unscheduled ID returns ErrNotFound.

type ScheduledID

type ScheduledID string

ScheduledID identifies a delayed start that has not begun yet. Engines such as Hatchet return it for a RunAt enqueue and hand out the RunID only once the run fires, so the two cannot share one field.

func (ScheduledID) IsZero

func (id ScheduledID) IsZero() bool

IsZero reports whether the ID is unset.

func (ScheduledID) String

func (id ScheduledID) String() string

String returns the ID as a string.

type SecretDetector

type SecretDetector interface {
	Detect(job Job) []SecretFinding
}

SecretDetector decides whether a job carries credentials (MS-05). The stock one is NewSecretDetector; an app with its own rules (a vault's token format, an entropy check) supplies its own.

func NewSecretDetector

func NewSecretDetector(r SecretRules) SecretDetector

NewSecretDetector returns the stock detector. It reports a JSON object key or label key that names a credential ("password", "api_key", "client_secret", "token", ...) holding a non-empty string, and any string value matching a known credential format: a PEM private key, an AWS access key ID, a Stripe live key, a GitHub or Slack token, a JWT. A value that is an object (a SecretRef) is never reported. A payload that is not JSON is scanned whole by the value rules only.

type SecretFinding

type SecretFinding struct {
	// Where locates it: "payload" plus a JSON path ("payload.user.password"),
	// "label <key>", or "chain[2].payload...".
	Where string

	// Rule names the rule that matched ("key", "pem-private-key",
	// "aws-access-key", ...), so a false positive can be allowed by name.
	Rule string
}

SecretFinding is one place a detector believes holds a credential.

type SecretRef

type SecretRef struct {
	Ref string `json:"secret_ref"`
}

SecretRef names a secret instead of carrying it: put it in a payload where the credential would go, and resolve it in the step with ResolveSecret. Its JSON form is {"secret_ref": "<name>"}, an object, so SecretGuard never reports it.

type SecretResolver

type SecretResolver interface {
	ResolveSecret(ctx context.Context, name string) ([]byte, error)
}

SecretResolver turns a secret's name into its value at the moment a step needs it: a vault, a cloud secret manager, environment variables.

type SecretRules

type SecretRules struct {
	// AllowKeys are JSON object keys and label keys never reported by the
	// key rule, compared after the same folding the rule applies
	// (lower case, "_", "-" and "." removed), such as "pagetoken".
	AllowKeys []string

	// AllowRules are value rules turned off by name, such as "jwt" for
	// an app that passes signed links.
	AllowRules []string
}

SecretRules tunes the stock detector.

type Source added in v0.1.1

type Source string

Source names what started a run. The library derives the sources below from labels it already writes (SourceOf); an app names its own ("api", "webhook", "slack", "admin") with WithSource. A Source passes ValidateSource: lower-case letters, digits and . _ - only.

const (
	// SourceSchedule: a schedule or a declared cron fired the run
	// (LabelSchedule).
	SourceSchedule Source = "schedule"

	// SourceScheduleManual: ScheduleTriggerer started a schedule's run by
	// hand (LabelTrigger = LabelTriggerManual).
	SourceScheduleManual Source = "schedule-manual"

	// SourceChild: a step spawned the run (LabelParentRun).
	SourceChild Source = "child"

	// SourceBatch: the run is a member of a batch (batch.LabelBatch).
	SourceBatch Source = "batch"

	// SourceEvent: an events.Bus subscription started the run
	// (events.LabelEvent).
	SourceEvent Source = "event"

	// SourceExternal: a run recorder found the run on the engine without
	// having seen it enqueued (an engine UI, its CLI, a crash window).
	SourceExternal Source = "external"
)

The sources SourceOf derives from the library's own labels, and the one a run recorder gives a run it found on the engine but never saw enqueued.

func LibrarySources added in v0.1.1

func LibrarySources() []Source

LibrarySources returns every source the library itself names, as a fresh slice. App sources are not in it.

func SourceOf added in v0.1.1

func SourceOf(labels map[string]string) (source Source, ref string)

SourceOf returns the source of a run from its labels, and the source reference when there is one. An explicit LabelSource wins. Otherwise it derives one from the labels the library stamps, most specific first: a manual trigger of a schedule, a schedule fire, a spawned child, a batch member, an event subscription. For a derived source the reference is the natural ID (the schedule, the parent run, the batch, the event name). It returns "" when nothing says where the run came from; a caller picks its own default (a run recorder's default source).

type SpawnOption

type SpawnOption func(*spawnConfig)

SpawnOption configures Spawn.

func SpawnAllowDuplicates

func SpawnAllowDuplicates() SpawnOption

SpawnAllowDuplicates lets Spawn run on a client without CapIdempotencyKey. The journal still keeps a retried step from enqueueing a job twice, except when the process dies between an enqueue and the journal's save of its RunRef; with this option that window can start a job twice.

type StackTracer

type StackTracer interface {
	error
	StackTrace() string
}

StackTracer is an error that carries a stack trace, such as the panic error of middleware.Recover. StackOf finds it anywhere in an error's chain.

type Status

type Status string

Status is where a run is.

The run state machine

A run is created StatusScheduled (Job.RunAt in the future, CapDelayedStart) or StatusQueued, and ends in exactly one terminal status. The legal moves, what triggers each and the capabilities that gate it are the Transitions table below; this is the same thing in prose.

scheduled -> queued       the start time arrives (CapDelayedStart)
queued    -> running      a worker claims a ready step
running   -> queued       the attempt is handed back (graceful stop, dead
                          worker, lease lost) or the next step is ready
running   -> waiting      a Wait or Sleep step parks (CapWaitForEvent,
                          CapDurableSleep) or a step Snoozes (CapSnooze)
waiting   -> queued       the event arrives or the sleep ends
waiting   -> running      the resumed step starts at once
running   -> retrying     an attempt failed and RetryPolicy.Next allows
                          another; RunInfo.NextAttemptAt says when
                          (reported only with CapRunStates; other drivers
                          show queued or running for the same moment)
retrying  -> running      the backoff elapsed and a worker claimed it
retrying  -> queued       the backoff elapsed, waiting for a worker
running   -> succeeded    every step succeeded
running   -> failed       an attempt failed and RetryPolicy.Next says stop
                          (Max, MaxErrors, Until, Permanent, NoRetryOn),
                          and RetryPolicy.DiscardOnExhaust is off
running   -> discarded    a step returned Discard (CapDiscard +
                          CapRunStates), or the same stop with
                          RetryPolicy.DiscardOnExhaust on
                          (CapDiscardExhausted + CapRunStates)
any non-terminal -> cancelled   Cancel or CancelAll
                          (CapCancel); a Discard on a driver with only
                          CapDiscard also ends cancelled (see EndStatus)
any non-terminal -> failed      the run went past Workflow.RunTimeout
                          (CapRunTimeout) or ForceFail (CapForceRun)
any non-terminal -> succeeded   ForceComplete (CapForceRun)
failed    -> queued       Retry re-opens the run from its failed step
                          (CapRetryRun); the only move out of a terminal
                          status. A discarded or cancelled run is not
                          re-opened: enqueue a new job instead.

The active statuses (queued, running, waiting, retrying) describe the run as a whole, so with parallel steps a run can move between any two of them as its steps progress (one step retrying while another waits). The Transitions table therefore lists every active-to-active pair that a driver with all of the gating capabilities can report; a driver without CapRunStates never reports retrying or discarded.

Failed versus discarded: failed is a failure an operator must see, so OnFailure runs for it. Discarded is a run dropped on purpose or given up on by policy: OnFailure does not run, the chain does not continue, the idempotency key is released, and the Err matches ErrDiscarded and ErrCancelled. A run exhausts into discarded only when its effective policy (job, step or workflow, see EffectiveRetriesFor) opted in with DiscardOnExhaust; the default is failed.

Code that switches on a Status must keep a default branch: drivers that declare CapRunStates report StatusRetrying and StatusDiscarded, which older drivers never did, and a later version may add more.

const (
	StatusScheduled Status = "scheduled" // a delayed start that has not begun
	StatusQueued    Status = "queued"    // waiting for a worker
	StatusRunning   Status = "running"   // a step is executing
	StatusWaiting   Status = "waiting"   // suspended on a wait or sleep step
	StatusSucceeded Status = "succeeded"
	StatusFailed    Status = "failed"
	StatusCancelled Status = "cancelled"

	// StatusRetrying: no step is executing and at least one step attempt
	// failed and waits on its retry backoff. RunInfo.NextAttemptAt says
	// when it runs again. Only drivers with CapRunStates report it; the
	// others report StatusQueued or StatusRunning for the same moment.
	StatusRetrying Status = "retrying"

	// StatusDiscarded is terminal: the run was dropped on purpose rather
	// than failed. Two things give it: a step returned Discard(err), or a
	// step stopped retrying (attempts ran out, MaxErrors or Until stopped
	// it, or a Permanent error) under a RetryPolicy with DiscardOnExhaust
	// set. Without that opt-in a run that runs out of retries ends
	// StatusFailed, because that is a failure an operator must see. Its
	// Err matches ErrDiscarded and ErrCancelled. Only drivers with
	// CapRunStates report it; the others report StatusCancelled (with
	// CapDiscard) or StatusFailed for the same run.
	StatusDiscarded Status = "discarded"
)

The run statuses.

func Await

func Await(ctx context.Context, c Client, ref RunRef, opts AwaitOptions) (Status, error)

Await blocks until the run is terminal and returns its status, or ctx's error (give ctx a deadline for a timeout). It polls Client.Status, so it works on every driver; it holds no worker, but it does hold the caller, so call it from a request handler or a CLI, never from a step (use AwaitChild there).

func CancelStatuses

func CancelStatuses(q RunQuery) []Status

CancelStatuses is the statuses CancelAll cancels: q.Statuses, or, when empty, the runs not yet started (StatusScheduled and StatusQueued): clearing a backlog must not cut short work that has already begun.

func EndStatus

func EndStatus(have Caps, err error) Status

EndStatus is the terminal status a driver records for a run whose step ended it with err, given the driver's capabilities: a discarded run (IsDiscarded) is StatusDiscarded with CapRunStates, StatusCancelled with CapDiscard alone, and StatusFailed otherwise (Discard falls back to Permanent, the direction that never hides a failure); any other error is StatusFailed. Every driver calls it, so none decides the mapping differently.

func (Status) CanTransitionTo

func (s Status) CanTransitionTo(to Status) bool

CanTransitionTo reports whether a run in s may be reported as to by a driver with every capability of some transition between them. It ignores capabilities (a driver without CapRunStates never reports retrying); use TransitionsFrom to see what gates each move.

func (Status) IsTerminal

func (s Status) IsTerminal() bool

IsTerminal reports whether the run has finished and will not change again, with one exception: Retry re-opens a failed run (CapRetryRun). An idempotency key is released once its run is terminal.

func (Status) TransitionsFrom

func (s Status) TransitionsFrom() []Transition

TransitionsFrom returns the moves that leave s: none for a terminal status except failed, which Retry re-opens.

func (Status) Valid

func (s Status) Valid() bool

Valid reports whether s is one of the Status constants, so a driver or an admin surface can refuse a status it read from storage or a request.

type Step

type Step struct {
	// Name is unique within the workflow; After refers to it.
	Name string

	// After names the parent steps that must succeed before this one
	// starts. Empty means the step starts with the run.
	After []string

	// Run is the code the step executes.
	Run Handler

	// Wait suspends the run until a matching Event or the deadline.
	// Needs CapWaitForEvent.
	Wait *WaitSpec

	// Sleep suspends the run for this long. Needs CapDurableSleep.
	Sleep time.Duration

	// Retry overrides the workflow default. Nil means the workflow's;
	// RetriesNone() means never retry. See EffectiveRetries.
	Retry *RetryPolicy

	// Timeout caps one attempt. Zero means the workflow default.
	Timeout time.Duration

	// HeartbeatTimeout fails an attempt that goes this long without
	// calling Heartbeat (counted from the attempt's start, then from each
	// Heartbeat), with an error matching ErrHeartbeatTimeout and
	// ErrTimeout; the step's retry policy then applies. It catches a
	// handler that hangs well before a long Timeout would. Zero means
	// none. Run steps only. Needs CapHeartbeat.
	HeartbeatTimeout time.Duration
}

Step is one node of a workflow. Exactly one of Run, Wait and Sleep is set; Validate rejects anything else.

func Choice

func Choice(c ChoiceSpec) ([]Step, error)

Choice expands c into ordinary steps, so it runs on every driver with no capability beyond what the steps need: a decide step ("<Name>.decide"), one step per branch ("<Name>.<branch>") and a join step named Name. Every branch step starts once the decide step is done; the branches not chosen return at once without calling their Run, and the join step returns the chosen branch's output unchanged. A run listing therefore shows every branch step as succeeded; the one that did work is the one whose Run was called.

Append the result to Workflow.Steps. It returns an error wrapping ErrInvalid when the spec is incomplete or a branch name repeats.

func (Step) Kind

func (s Step) Kind() StepKind

Kind reports which kind the step is, or "" when none or several are set (an invalid step).

type StepCanceller

type StepCanceller interface {
	CancelStep(ctx context.Context, ref RunRef, step string) error
}

StepCanceller withdraws one step of a run, leaving the rest of the run to decide what happens next.

CancelStep acts on a step that is running, waiting on its retry backoff, snoozed, or a wait or sleep step that is suspended:

  • a running attempt's ctx is cancelled with cause ErrStepCancelled (context.Cause). Whatever the handler then returns decides: an output means the step succeeded and the run continues (a handler can return a fallback, the "interrupting boundary" of a process model); an error fails the step for good, without retry, with an error matching ErrStepCancelled, and the run fails as any failed run does (OnFailure runs);
  • a step with no attempt running (backoff, snooze, wait, sleep) fails for good at once with an error matching ErrStepCancelled.

A step that has not started, or has ended, returns ErrWrongState; an unknown run or step ErrNotFound. Cancelling the whole run is Cancel.

type StepEvent added in v0.1.1

type StepEvent struct {
	// Message is a short note for a person.
	Message string

	// Attrs are the event's key/value pairs, values already rendered as
	// strings so every sink stores the same text. Order is the caller's.
	Attrs []EventAttr

	// Level is the slog level the caller gave (slog.LevelInfo by default
	// with RecordEvent, any level with RecordEventLevel).
	Level slog.Level

	// At is when it was recorded. RecordEvent leaves it zero, because the
	// core reads no wall clock on a step's path (a fake clock in a test
	// must see the same time a sink stores); each sink stamps it from its
	// own clock when zero.
	At time.Time

	// Step is the StepInfo of the attempt that recorded it; zero outside
	// a driver-run step.
	Step StepInfo
}

StepEvent is one line of a step's timeline ("downloaded 12 MB, 4,000 rows"), recorded from inside a step with RecordEvent and handed to every StepEventSink bound on the step's ctx: a run recorder stores it with the step, a tracer adds it as a span event, a log capture writes it.

type StepEventFunc added in v0.1.1

type StepEventFunc func(ctx context.Context, e StepEvent)

StepEventFunc adapts a function to StepEventSink.

func (StepEventFunc) RecordStepEvent added in v0.1.1

func (f StepEventFunc) RecordStepEvent(ctx context.Context, e StepEvent)

RecordStepEvent calls f.

type StepEventSink added in v0.1.1

type StepEventSink interface {
	RecordStepEvent(ctx context.Context, e StepEvent)
}

StepEventSink receives the events a step records. A sink must be safe for concurrent use (steps of one run may run in parallel) and must not block: RecordEvent calls it on the step's goroutine.

type StepInfo

type StepInfo struct {
	// Workflow is the workflow name.
	Workflow string

	// Step is the step name.
	Step string

	// RunID is the run the step belongs to.
	RunID RunID

	// Attempt is 1 for the first attempt and grows by one per retry.
	Attempt int

	// Labels are the run's Job.Labels. Drivers pass the run's own map;
	// handlers and middleware must not modify it.
	Labels map[string]string

	// Queue is the run's queue (EffectiveQueue). Drivers without
	// CapQueue set DefaultQueue or leave it empty.
	Queue string

	// ReadyAt is when this attempt became ready to claim, on the
	// driver's clock: the run's admission for a first step, its parents'
	// end for a later one, the backoff's end for a retry. Now minus
	// ReadyAt is how long the attempt waited for a worker (the queue wait
	// a metric reports). Zero when the driver does not know it.
	ReadyAt time.Time

	// Tenant is the run's Job.Tenant; empty without one or on a driver
	// without CapTenant.
	Tenant string

	// Node is the identity of the node running this attempt (the
	// driver's node-ID option, DefaultNodeID when unset); empty on a
	// driver without CapNodeIdentity.
	Node string
}

StepInfo describes the step attempt a Handler is running, so middleware (logging, tracing, metrics) and handlers can tell runs and attempts apart without a driver-specific context. Every driver sets it on the ctx it passes to a step handler, with WithStepInfo, before the Middleware chain runs.

func StepInfoFrom

func StepInfoFrom(ctx context.Context) (StepInfo, bool)

StepInfoFrom returns the StepInfo a driver set on ctx, and false when ctx is not a step's context (for example in an enqueue middleware).

func (StepInfo) TickAt

func (i StepInfo) TickAt() (time.Time, bool)

TickAt returns the tick this run was fired for (see LabelTickAt); false when no schedule fired it.

type StepKind

type StepKind string

StepKind is what a step does. Exactly one kind is set per step.

const (
	StepRun   StepKind = "run"   // calls Step.Run
	StepWait  StepKind = "wait"  // suspends until an Event matches Step.Wait
	StepSleep StepKind = "sleep" // suspends for Step.Sleep
)

The step kinds.

type StepMiddlewareProvider

type StepMiddlewareProvider interface {
	StepMiddleware() []Middleware
}

StepMiddlewareProvider contributes step middleware. It wraps every step's Run and the workflow's OnFailure for workflows registered through the client Use returns, inside any middleware the driver itself was built with (so a driver-level codec has already opened the input). The first entry is the outermost.

type StopCauser

type StopCauser interface {
	StopCause() error
}

StopCauser reports why a stopped client stopped. A driver with recycling implements it; Forward, failover and the middleware client wrappers pass it through.

StopCause returns nil while the client runs and after a stop that was asked for (the Start ctx ended, Drain, a stop-when-idle option), and a *RecycleError when a Recycle limit stopped it. It is final once the Drainer's Done channel is closed.

type Tick

type Tick struct {
	// ScheduleID is the schedule. For a declared Workflow.Cron entry it
	// is the driver's stable id for that declaration, the same one
	// ScheduleLister reports.
	ScheduleID ScheduleID

	// Name is the Schedule.Name; for a declared cron, the expression.
	Name string

	// Workflow is the workflow the schedule runs.
	Workflow string

	// Declared is true for a Workflow.Cron entry.
	Declared bool

	// At is the tick's time; for TickMisfired, the newest dropped tick.
	At time.Time

	// Location is the zone the schedule is evaluated in.
	Location *time.Location

	// Outcome is what happened.
	Outcome TickOutcome

	// Reason is set for TickSkipped: TickReasonWhen, TickReasonWindow or
	// TickReasonPaused.
	Reason string

	// Missed is how many ticks a TickMisfired dropped.
	Missed int

	// Run is the run a TickFired enqueued.
	Run RunRef

	// Err is why a TickFired could not enqueue.
	Err error

	// Labels are the schedule's labels. Observers must not modify them.
	Labels map[string]string

	// Manual is true for a TickFired that ScheduleTriggerer started by
	// hand; At is then the driver's clock at the trigger.
	Manual bool
}

Tick reports one tick of a schedule, or, for TickMisfired, one batch of dropped ticks.

type TickNotifier

type TickNotifier interface {
	OnTick(fn TickObserver) error
}

TickNotifier is an optional interface a client implements to report every tick outcome: fired, skipped and misfired. A client implements it when it declares CapTickEvents. The contract is RunNotifier's: register before Start (ErrAlreadyStarted after), observers run on a driver goroutine in registration order, panics are recovered, delivery is at least once and may be lost on a crash. Ticks are reported only by the node that fired or skipped them.

type TickObserver

type TickObserver func(ctx context.Context, t Tick)

TickObserver observes ticks. It must be idempotent, because delivery is at least once.

type TickOutcome

type TickOutcome string

TickOutcome is what happened to one schedule tick.

const (
	// TickFired: the tick enqueued a run (Tick.Run), or tried to and the
	// enqueue failed (Tick.Err).
	TickFired TickOutcome = "fired"

	// TickSkipped: the tick was due and on time but did not fire;
	// Tick.Reason says why.
	TickSkipped TickOutcome = "skipped"

	// TickMisfired: the scheduler woke late and its misfire policy
	// dropped Tick.Missed ticks, which a scheduler that looks only at the
	// current minute cannot even see.
	TickMisfired TickOutcome = "misfired"
)

The tick outcomes.

type TickSource

type TickSource interface {
	// Between returns the fire times t with from < t <= to, ascending, at
	// most limit of them.
	Between(from, to time.Time, loc *time.Location, limit int) []time.Time
}

TickSource lists a schedule's fire times. cron.Schedule satisfies it; the core names only the one method it needs, so it does not import a parser.

type Transition

type Transition struct {
	From, To Status

	// Trigger says what causes the move, in a few words.
	Trigger string

	// Needs is every capability the driver must declare for the move to be
	// reported; zero means any driver.
	Needs Caps
}

Transition is one legal move of the run state machine: a run in From may be reported as To when Trigger happens on a driver that has every capability in Needs.

func Transitions

func Transitions() []Transition

Transitions returns every legal move of the run state machine. The slice is a copy. Drivers and tests use it to check that a status change they are about to store (or one they read) is a move the machine allows.

type UnsupportedError

type UnsupportedError struct {
	// Field is the spec field or feature, such as "Job.RunAt" or
	// "Schedule.Location".
	Field string

	// Missing is the capabilities that would be needed, when the refusal
	// comes from a capability check; zero otherwise.
	Missing Caps

	// Reason is an optional detail, such as the workflow name.
	Reason string
}

UnsupportedError names what a driver cannot do. It matches ErrUnsupported under errors.Is.

func (*UnsupportedError) Error

func (e *UnsupportedError) Error() string

Error names the field, the missing capabilities and the reason, so the message alone tells an operator what to change.

func (*UnsupportedError) Is

func (e *UnsupportedError) Is(target error) bool

Is makes errors.Is(err, ErrUnsupported) true.

type Upcaster

type Upcaster func(raw []byte) ([]byte, error)

Upcaster turns a payload of one version into the next version's shape (version n to n+1): rename a field, fill a default, split a value.

type UpstreamLimit

type UpstreamLimit struct {
	// contains filtered or unexported fields
}

UpstreamLimit is a leaky-bucket, cost-based rate limit on calls a step makes to an outside API (PS-02), one bucket per key. Unlike Workflow.RateLimits, which counts run starts, it counts what the upstream counts: the cost of each call, against a bucket that drains over time. Build it with NewUpstreamLimit.

func NewUpstreamLimit

func NewUpstreamLimit(store BucketStore, spec BucketSpec, clk clock.Clock) (*UpstreamLimit, error)

NewUpstreamLimit returns a limit over store with spec, reading time from clk. A nil store means a new MemoryBuckets; a nil clk means clock.Real. It returns an error wrapping ErrInvalid when spec is not positive and finite.

func (*UpstreamLimit) Observe

func (u *UpstreamLimit) Observe(ctx context.Context, key string, level float64) error

Observe records the level the upstream reported for key, so the next Take decides from the upstream's count. Call it with each response's rate-limit header.

func (*UpstreamLimit) Take

func (u *UpstreamLimit) Take(ctx context.Context, key string, cost float64) error

Take reserves cost units in key's bucket. It returns nil when they fit; otherwise RetryAfter(ErrThrottled, wait), which a step returns as it is so the attempt is retried once the bucket has room (see ErrThrottled). A cost above the bucket's capacity can never fit and is a Permanent error wrapping ErrInvalid; a negative cost is refused the same way.

func (*UpstreamLimit) TakeOrSnooze

func (u *UpstreamLimit) TakeOrSnooze(ctx context.Context, key string, cost float64) error

TakeOrSnooze is Take for a driver with CapSnooze: when the bucket has no room it returns Snooze(wait), which puts the step back without spending an attempt, since a spent upstream limit is "not yet", not a failure. On a driver without CapSnooze a snooze is an ordinary error and spends an attempt; use Take there.

func (*UpstreamLimit) Wait

func (u *UpstreamLimit) Wait(ctx context.Context, key string, cost float64) (time.Duration, error)

Wait is Take without the error: it reserves cost and returns zero, or reserves nothing and returns how long until cost fits, for a caller that would rather sleep in the step than retry it.

type WaitSpec

type WaitSpec struct {
	// Event is the event name to wait for.
	Event string

	// Key is a fixed key, the same for every run of the workflow. Empty
	// matches an Event with an empty key. Use KeyFrom for a key that
	// differs per run.
	Key string

	// KeyFrom derives the key per run, so many runs of one workflow can
	// wait on the same event name for different things ("payment.succeeded
	// for THIS order"). The driver evaluates it once, when the wait step
	// begins, against the run's Job (its original Payload and Labels, not
	// the step's input), and Signal matches Event.Key against the result.
	// Func is evaluated in Go by any driver (ResolveKey does it); Expr is
	// an expression in the engine's language over the run input (Hatchet's
	// CEL), and a driver without one refuses it with an UnsupportedError.
	// At most one of Key and KeyFrom is set.
	KeyFrom ConcurrencyKey

	// Deadline is the absolute time the wait fails.
	Deadline time.Time

	// Timeout is how long the wait lasts from when it begins.
	Timeout time.Duration
}

WaitSpec is a durable wait: the run releases its worker until Signal delivers an Event with the same name and key, or the wait expires. At most one of Deadline and Timeout is set; neither means wait forever.

func (WaitSpec) ResolveKey

func (w WaitSpec) ResolveKey(job Job) (string, error)

ResolveKey returns the key a run waits on: KeyFrom.Func applied to job when set, else the fixed Key. A KeyFrom.Expr is the engine's to evaluate, so here it returns an UnsupportedError naming the field; a driver that evaluates expressions handles that case before calling.

type Window

type Window struct {
	From, To string
}

Window is a time-of-day range on the wall clock of a schedule's zone, "HH:MM" each (24-hour). From is inclusive and To exclusive; a To before From wraps midnight ("22:00" to "06:00" is the night). From equal to To is refused, because it would mean either nothing or the whole day.

type Windows

type Windows struct {
	Between []Window
	Unless  []Window
}

Windows limits a schedule's ticks by time of day: a tick fires only inside at least one Between window (no Between windows: at any time) and outside every Unless window.

func (Windows) Allows

func (w Windows) Allows(t time.Time, loc *time.Location) bool

Allows reports whether t, read on loc's wall clock (UTC when loc is nil), passes w. Windows that fail validation never match. It is a pure function, so every driver applies windows the same way.

func (Windows) IsZero

func (w Windows) IsZero() bool

IsZero reports whether w limits nothing.

type WorkerScaler

type WorkerScaler interface {
	// QueueLoads returns the load of every configured queue, sorted by
	// name. Before Start it returns ErrNotStarted.
	QueueLoads(ctx context.Context) ([]QueueLoad, error)

	// SetWorkers sets queue's worker count to n, effective for the next
	// attempt claimed: lowering it never interrupts running attempts.
	// n below one wraps ErrInvalid; a queue the client was not configured
	// with wraps ErrNotFound. Before Start it returns ErrNotStarted.
	SetWorkers(ctx context.Context, queue string, n int) error
}

WorkerScaler is implemented by a client whose per-queue worker counts can change while it runs. A client implements it when it declares CapWorkerScale. It covers the queues the client was configured with (its QueueConfig list); the counts are this client's own, so on a shared store each process balances its own workers.

type Workflow

type Workflow struct {
	// Name is the key Job.Workflow and Schedule.Workflow refer to.
	Name string

	// Steps form a DAG through Step.After. A step with no After starts
	// first; a plain list with no After runs its steps in parallel, so
	// chain them with After for sequence.
	Steps []Step

	// Cron lists 5-field expressions declared with the workflow and
	// evaluated in CronLocation. They are replaced, never duplicated, when
	// the workflow registers again. Needs CapCron.
	Cron []string

	// CronLocation is the time zone Cron is evaluated in. Nil means UTC
	// (the behaviour before the field existed); time.Local is refused, so
	// no declared cron runs in the host's implicit zone. Any zone but UTC
	// needs CapCronZone: a UTC-only engine refuses it rather than shifting
	// the expression, which would break across DST.
	CronLocation *time.Location

	// Misfire says what happens to Cron ticks the driver reached late or
	// not at all (nothing was running). The zero value is the driver's
	// default, documented in its package. Setting it without Cron is
	// invalid. See Misfire for the capability each policy needs.
	Misfire Misfire

	// Concurrency caps simultaneous runs per key. Nil means no cap.
	// Needs CapConcurrencyKey.
	Concurrency *Concurrency

	// RateLimits cap how often runs may start. Needs CapRateLimit.
	RateLimits []RateLimit

	// Idempotency says how Job.IdempotencyKey is treated. Empty means
	// IdempotencyOptional.
	Idempotency IdempotencyPolicy

	// Retry is the default for steps whose Retry is nil. Its zero value
	// means no retries.
	Retry RetryPolicy

	// Timeout is the default per-step timeout. Zero means none.
	Timeout time.Duration

	// OnFailure runs once when a run fails for good, with the failing
	// step's input. Nil means nothing runs.
	OnFailure Handler

	// Queue is the default queue of this workflow's runs; Job.Queue
	// overrides it per run. Empty means DefaultQueue. A queue other than
	// DefaultQueue needs CapQueue. See QueueConfig for how workers serve
	// queues.
	Queue string

	// ScheduleWhen is asked before each tick of this workflow's schedules
	// fires: the declared Cron and every Client.Schedule of the workflow
	// alike (switch on tick.Name for a rule per schedule). Returning false
	// skips the tick, which tick observers see as TickSkipped with Reason
	// TickReasonWhen. Nil fires every tick. It lives on the workflow, not
	// on Schedule, because a condition is code that every node registers,
	// while a durable driver stores a Schedule as data and may fire it on
	// any node. It is called on the firing node with the driver's context
	// and must return quickly. Needs CapScheduleWhen.
	ScheduleWhen func(ctx context.Context, tick Tick) bool

	// CronWindows limits the declared Cron ticks to time-of-day windows on
	// CronLocation's wall clock. Zero means no limit. Needs
	// CapScheduleWindow.
	CronWindows Windows

	// CronOverlap is the overlap policy of every Cron entry: OverlapSkip
	// skips a tick while that entry's previous run is unfinished. The
	// zero value is OverlapAllow. Each entry is its own schedule, so one
	// entry's run does not hold back another's tick. It needs Cron.
	CronOverlap OverlapPolicy

	// CronSeconds says every Cron entry has six fields, seconds first
	// (cron.ParseSeconds). It is never guessed from the field count,
	// because six fields could as well mean a trailing year. Needs Cron
	// and CapCronSeconds.
	CronSeconds bool

	// CronActive limits the declared Cron to a period: ticks before
	// Active.From or from Active.Until on do not fire (see Active). Zero
	// means always. Needs Cron and CapScheduleActive.
	CronActive Active

	// CronKeepRuns keeps only the newest N finished runs fired by each
	// Cron entry; older finished runs are forgotten as RunDeleter would.
	// Zero keeps them all (the driver's retention still applies). Needs
	// Cron and CapScheduleKeep.
	CronKeepRuns int

	// RunTimeout caps a run's whole life, from its admission to its end
	// (waits and sleeps included). A run past it is ended: running
	// attempts see their ctx cancelled with cause ErrRunTimeout, and the
	// run fails with an error matching ErrRunTimeout and ErrTimeout, which
	// is never retried. Zero means none. Needs CapRunTimeout.
	RunTimeout time.Duration

	// OnTimeout runs once, in place of OnFailure, when a run fails for
	// good because of a timeout (IsTimeout: a step attempt past its
	// Timeout on its last attempt, a wait past its deadline, a heartbeat
	// timeout or RunTimeout), with the failing step's input (the run's
	// payload for RunTimeout outside any step). Nil means OnFailure runs
	// for timeouts too. Needs CapRunTimeout.
	OnTimeout Handler
}

Workflow is a named graph of steps plus the policies that apply to every run of it. A one-step workflow is a plain background job.

Directories

Path Synopsis
admin module
Package batch runs a group of jobs as one unit with Then, Catch and Finally callbacks, on any workflow.Client.
Package batch runs a group of jobs as one unit with Then, Catch and Finally callbacks, on any workflow.Client.
cli module
Package clock is the time source every scheduling decision reads.
Package clock is the time source every scheduling decision reads.
Package cron parses cron expressions and computes their fire times.
Package cron parses cron expressions and computes their fire times.
tzdata
Package tzdata embeds the IANA time zone database in the binary, for apps that run schedules in a container image without one (scratch, distroless and many slim images ship no /usr/share/zoneinfo).
Package tzdata embeds the IANA time zone database in the binary, for apps that run schedules in a container image without one (scratch, distroless and many slim images ship no /usr/share/zoneinfo).
Package events starts runs from events: publish/subscribe inside the library (EV-01) and an inbound webhook handler that verifies a signature and enqueues (EV-03), with HMAC signing schemes for both directions (WH-03).
Package events starts runs from events: publish/subscribe inside the library (EV-01) and an inbound webhook handler that verifies a signature and enqueues (EV-03), with HMAC signing schemes for both directions (WH-03).
Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work.
Package failover joins two workflow clients into one: enqueues go to a primary backend, and when the primary is unavailable they go to a secondary, so a job is still accepted while (say) the Hatchet engine is down and a local SQL store takes the work.
hatchet module
Package hybrid splits one workflow.Client across two backends: every schedule fires on a scheduler client (usually the in-process memory driver) while every run lives on a jobs client (a durable driver), the common "cron in the process, jobs in the database" shape.
Package hybrid splits one workflow.Client across two backends: every schedule fires on a scheduler client (usually the in-process memory driver) while every run lives on a jobs client (a durable driver), the common "cron in the process, jobs in the database" shape.
Package memory is the in-process driver of workflow.Client: runs, waits, sleeps, schedules and queues live in maps and goroutines inside the client value, with no database and no server.
Package memory is the in-process driver of workflow.Client: runs, waits, sleeps, schedules and queues live in maps and goroutines inside the client value, with no database and no server.
Package middleware is the app-side layer that sits above every driver: the conveniences a job queue offers on top of its broker, written once for Go.
Package middleware is the app-side layer that sits above every driver: the conveniences a job queue offers on top of its broker, written once for Go.
otel module
recorder module
sqlrecorder module
redis module
sentry module
sqlstore module
pgnotify module
pgxtx module
Package workflowtest holds the conformance suite every workflow driver runs, and two doubles for app tests: Fake, a recording client that runs nothing and refuses what a real driver refuses, with expressive assertions (AssertEnqueued and its siblings); and Recorder, which wraps a real client and records its enqueues.
Package workflowtest holds the conformance suite every workflow driver runs, and two doubles for app tests: Fake, a recording client that runs nothing and refuses what a real driver refuses, with expressive assertions (AssertEnqueued and its siblings); and Recorder, which wraps a real client and records its enqueues.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL