temporaless

README

Temporaless

A storage-first, language-agnostic, serverless workflow framework for Go, Python, and Rust (storage layer).

The core idea: every workflow boundary (start, activity, durable timer, signal, claim) is a protobuf record at a deterministic path in object storage, keyed by a caller-supplied id. When workflow code reaches a stored boundary, the runtime first looks for a matching record. If one exists, the stored result is reused. If not, the boundary is executed and stored. There is no engine, no control plane, no central server — the storage backend is the source of truth, and processes are interchangeable.

Not a Temporal replacement. A narrower framework for data workflows where activities are mostly fetch / normalize / persist and storage can be the durable coordination point.

• docs/getting-started.md — single-page walkthrough: store, workflow, retries, sleep, events, schedule, inspect, sweep
• docs/deployment.md — production patterns (S3/GCS, ConnectRPC, multi-process, multi-region)

Layout

api/                  protobuf API definitions only
core/{go,py}/         Runtime + generated protobuf + OpenDAL store + ConnectRPC service
core/rs/              Rust SDK — storage layer only (native opendal crate); runtime layers TBD
adapters/{go,py}/     Adapters: claims, schedulers, inspectors, retention, Temporal compat
examples/{go,py}/     Runnable demos: fetch-prices, llm-completion, production-server, quant-service, stocks-pipeline, twitter-webhook
docs/                 Architecture and design notes

Adapters

Adapter	Purpose
adapters/go/connectstore	ConnectRPC store adapter; v2 regenerated-stub parity is a 0.3 follow-up
adapters/go/gocdkclaims	Create-only activity claims via GoCDK Blob IfNotExist (S3, GCS native atomicity)
adapters/go/temporalcompat	Run Temporaless-shaped handlers on the real Temporal Go SDK (worker direction)
adapters/py/temporalcompat	Same for Python via temporalio
adapters/py/prefectcompat	Run Temporaless-shaped handlers as Prefect 3 flows / tasks; keep Prefect's UI + scheduling, keep our storage-first replay
adapters/py/indexstore	Optional SQLite query index for workflow/activity listing, retention sweeps, and indexed due-timer queries
adapters/go/timerscanner	Find due durable timers belonging to in-flight workflows
adapters/go/cronscheduler	In-process cron scheduler with stateless seeding from existing runs
adapters/go/inspector	List in-flight / failed workflows, reset records for re-execution
adapters/go/janitor	Sweep COMPLETED runs older than a max-age threshold
adapters/go/backfill	Run a workflow over many run_ids with bounded concurrency + per-run status (Dagster/Prefect/Airflow-style backfill)
adapters/go/dependencies	Cross-pipeline durable wait — WaitForWorkflow(store, key, newResult) returns upstream's result or a typed pending/failed error
adapters/go/dispatch	Fire-and-forget goroutine pool for gRPC-shaped handlers — DoAsync(method, req) + Shutdown(ctx) that drains in-flight goroutines (default 15s) before cancelling. In-process only; pair with workflows when you need durability.

Python equivalents for the operations adapters live in core/py/src/temporaless/{timerscanner,cronscheduler,inspector,janitor,backfill,dependencies,dispatch}.py (no separate uv project — they have no third-party deps). backfill runs a workflow over many run_ids with bounded concurrency; dependencies.wait_for_workflow is the cross-pipeline durable wait primitive; dispatch.Dispatcher is the fire-and-forget asyncio-task pool that mirrors adapters/go/dispatch. Rust users get the same shape via temporaless::dispatch.

Examples

Example	Demonstrates
examples/go/fetch-prices / examples/py/fetch_prices.py	Hello-world workflow with one activity
examples/go/llm-completion / examples/py/llm_completion.py	Retry policy + annotations + replay
examples/go/stocks-pipeline	Cron scheduler + multi-activity workflow + replay
examples/go/twitter-webhook	WaitEvent + external SendEvent, workflow stays IN_PROGRESS until signal arrives
examples/py/quant_signals.py	asyncio.gather over 8 parallel symbol fetches + serial signal compose, full replay
examples/go/quant-service / examples/py/quant_service.py	Canonical ConnectRPC service: methods with connect.Request/connect.Response shape wrapped by workflow.HandleConnect (Go) or @wrap_workflow_method (Python) — production deploys mount this on any ConnectRPC mux / asgi_application
examples/py/stocks_cron.py	Cron-driven workflow: scheduler dispatches per-symbol workflows, statelessly seeded from existing run records
examples/py/approval_workflow.py	Long-running: durable sleep + wait-for-event + multi-step replay across simulated process deaths
examples/py/data_pipeline.py	Airflow-style ETL: extract → parallel transform (fan-out) → validate → conditional branch → load → notify, with backfill and replay
examples/go/production-server / examples/py/production_server.py	Production server: Python is current for v2 storage; Go regenerated-stub parity is a 0.3 follow-up. Shows ConnectStore + bearer-token auth interceptor + /healthz & /readyz endpoints + structured JSON logs with correlation IDs + graceful shutdown on SIGTERM.

Docs

• docs/philosophy.md — design tenets in one page (read first)
• docs/comparisons.md — honest comparison vs Temporal / n8n / Prefect / Dagster
• docs/getting-started.md — single-page walkthrough
• docs/deployment.md — production deployment patterns
• docs/production-checklist.md — pre-launch checklist (storage, ConnectStore, workflow service, operators, observability, failure modes)
• docs/runbook.md — operator runbook for common incidents (stuck workflows, claim leaks, storage outages, DR)
• docs/architecture.md — goals and core model
• docs/storage-rpc.md — RecordStoreService contract
• docs/scheduling.md — durable timers, cron, scanner, distribution
• docs/claims.md — claim coordination tiers
• docs/hard-cases.md — concurrency, retries, side effects, backend atomicity
• docs/adapter-contract.md — what adapters must declare
• docs/temporal-adapter.md — strict compatibility position
• docs/orchestrator-adapters.md — Dagster / Prefect adapter notes
• docs/dependencies.md — what lives where (Flox vs go.mod vs uv)
• docs/benchmarks.md — Go and Python benchmark suites with cross-language baseline numbers
• docs/analytics.md — bucket archive, optional query index, and offline protobuf analytics
• docs/sdks.md — cross-SDK surface comparison + capability matrix (Go / Python / Rust)

Development

flox activate
flox activate -- scripts/check       # proto + Python core/adapters gate
TEMPORALESS_CHECK_GO_RUST=1 flox activate -- scripts/check  # include explicitly gated Go/Rust parity checks
flox activate -- scripts/bench-go    # Go benchmarks (storage + workflow hot paths)
flox activate -- scripts/bench-py    # Python benchmarks (same suite, same output format)

The Flox manifest intentionally stays thin: just go, python313, uv, buf, plus the C runtime libs OpenDAL and Protovalidate need. Everything else lives in go.mod, core/py/uv.lock, or Buf remote plugin config.

Storage convention

Records are stored as protobuf binary at deterministic v2 keys:

temporaless/v2/{namespace}/{workflow_id}/{run_id}/workflow.binpb
temporaless/v2/{namespace}/{workflow_id}/{run_id}/activity/{activity_id}.binpb
temporaless/v2/{namespace}/{workflow_id}/{run_id}/timer/{timer_id}.binpb
temporaless/v2/{namespace}/{workflow_id}/{run_id}/event/{event_id}.binpb
temporaless/v2/{namespace}/{workflow_id}/{run_id}/claim/{claim_id}.binpb

Keys are constructed from protobuf keys; runtime code never parses paths back into record identity. A run's records stay co-located under one prefix so replay prefetch, run deletion, and bucket lifecycle rules remain simple.

IDs may contain letters, numbers, ., _, -, :, and =. Slashes are rejected because object keys are path-like. Namespace and workflow ID values beginning with _ are reserved for Temporaless system prefixes such as _latest and _due. The framework does not generate IDs — workflow IDs, run IDs, activity IDs, timer IDs, claim owner IDs, and event IDs are application-owned and must be passed explicitly.

The core storage contract is generated from temporaless.v1.RecordStoreService: point GET/PUT/DELETE, run-scoped lists for replay prefetch and run deletion, create-if-absent claims, latest-run pointers, and the compact due-timer ledger. Cross-run search lives on optional RecordQueryService implementations such as temporaless-indexstore; the bucket remains the source of truth and the index is rebuildable.

What you do and do not get

Generic core (in scope): workflow + activity replay, retry policy with RETRYING-record persistence, durable timers and signals via EventRecord, claim-based coordination tiers, durable structured annotations, in-process scheduler with O(1) latest-run pointer seeding, run-scoped prefetch/deletion, and comprehensive proto-typed errors.

Temporal-flavored knobs (adapter-only or out of scope): activity timeouts, heartbeats, sticky task queues, signal-channel select, workflow-level retry policy, child workflows, payload converters. Use adapters/{go,py}/temporalcompat if you need any of those.

Search and retention (optional index): bucket-only deployments run workflows, scheduling, durable timers, and lifecycle-based retention without a database. Listing workflows, inspector views, and indexed sweeps require a derived query index or an offline scan.

Stateful processes (none required): the core runtime is a pure function of (input, store_state). The cron scheduler keeps last-fires in memory but exposes Snapshot()/Restore() for explicit migration, plus LastFireFromRuns for deriving state from latest-run pointer objects when run_ids embed fire times. Timer resumption uses a compact due ledger, not a full bucket walk.

Python is async-only. Workflow bodies, activity bodies, and the runtime entry points (run, execute_activity, wait_event, sleep) are all async def. Sync callables are rejected at wrap time. The framework's I/O-bound workloads (LLM, HTTP, vendor APIs) are a natural fit for async; aligning with the Temporal Python SDK removes impedance. Go stays sync — goroutines + sync function signatures are idiomatic Go and there is no equivalent of async/await.

License

Apache 2.0 — see LICENSE for the full text.