engine

package
v0.0.0-...-606a6a1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Overview

Package engine is the durable execution driver: it runs a compiled workflow specification as a Temporal workflow, so a run survives process crashes, deploys, and waits measured in days.

It is one of two drivers over one execution model. The other is the local interpreter in the parent package (RunWithInputs), which executes the same specification in-process with no Temporal at all. Both dispatch through the same step executor, and anything observable that differs between them is a defect, because local runs exist to tell an author what production will do. Shared behavior cases live in pkg/flowstate/v1/internal/conformance and both drivers run them; add a case there rather than in one driver's own tests, and check that both drivers actually call the set it joins. A value with one meaning belongs in the parent package, which both drivers import, so one constant cannot disagree with itself.

The package sits between two boundaries. Below it are the generated flowstate.v1 types: the specification it executes, the RunState it suspends into, and the outputs it reports are all schema messages, and nothing here redefines their shape. Above it is the Temporal SDK: Register registers Run as the single pinned workflow type together with its activities, and everything Temporal already does well (timers, retries, signals, Continue-As-New, schedules) is surfaced rather than reimplemented.

The invariants below constrain every change here. They are stated fully, with their reasons, in docs/ARCHITECTURE.md; read that before a structural change, because a change that violates one is a bug even when the tests pass.

  • Workflow-side code is pure and pinned. Run executes under replay, so anything nondeterministic, version-sensitive, or I/O-bound belongs in an activity, never in workflow code. CEL evaluation is the one deliberate exception, accepted because the interpreter is version-pinned per run: a run finishes on the interpreter it started on and takes the current version only at Continue-As-New.
  • The workflow's own vars are an activity (WorkflowVars), and not for the reason above: Continue-As-New is the one seam replay does not cover, so an inline evaluation there could change value mid-run.
  • RunState is a wire contract between interpreter versions. One version writes it at Continue-As-New and a different version reads it back, so it obeys the rules a published message obeys.
  • Secrets never enter workflow history. A secret crosses this package only as a reference; the activity that needs the value resolves it through the capabilities in TaskRuntimeConfig, and errors are scrubbed before Temporal can persist them as failures.
  • A run that cannot continue must fail, not hang. RunState is weighed before suspending (v1.CheckRunStateSize), because a payload past Temporal's blob limit fails the workflow task, which is retried forever in silence.

Index

Constants

View Source
const ProgressQuery = "flowstate.progress"

ProgressQuery is the query name a client asks for a run's position by.

Namespaced, because a query name is a public identifier on every workflow this engine runs and Temporal's own tooling puts its built-ins (`__stack_trace`, `__enhanced_stack_trace`) in the same namespace.

View Source
const RunTaskQueueName = "flowstate-run-task-queue"

RunTaskQueueName is the task queue every run is submitted to when a deployment configures no per-tenant routing.

It is also the default `flow worker --task-queue` polls, which is what makes a first run against `temporal server start-dev` need no configuration at all (invariant 8). A deployment that routes tenants to their own queues sets TaskQueues.Prefix instead, and this name then addresses nothing — see TaskQueues.

Deliberately free of [taskQueueSeparator]. Every composed queue name contains one, so no tenant's queue can ever collide with this one, whatever prefix an operator picks. Asserted by TestRunTaskQueueNameCannotBeComposed.

View Source
const StateQuery = "flowstate.state"

StateQuery is the query name a client asks for a run's carried state by — its top-level `vars:` and what every currently active `loop:` is carrying between iterations.

A second query beside ProgressQuery rather than a field added to it: the two answer different questions at different costs. Position is small and answerable from the moment a run starts; state can be as large as an author's own `vars:` and loop bindings, which is why it carries its own bound (see [entityStateMaxLoopEntries] and [entityStateMaxBytes]) instead of inheriting one sized for a step id and a short path.

This is the answer to a gap ProgressQuery cannot close on its own: an entity — a run shaped as `loop:` + `wait_for_signal:`, never meant to reach STATUS_COMPLETED — is by design always RUNNING, and ProgressQuery answers only where such a run is, never what it holds. Namespaced for ProgressQuery's own reason: a query name is a public identifier on every workflow this engine runs.

Variables

This section is empty.

Functions

func DeploymentOptions

func DeploymentOptions(deployment, buildID string) (worker.DeploymentOptions, error)

DeploymentOptions builds the worker's versioning configuration.

Returned rather than applied so the caller can see what it is opting into, and so the zero value — versioning off — is what an unconfigured deployment gets. Both a deployment name and a build ID are required to turn it on: a version is the pair, and honouring half of it would produce a worker that claims a version it cannot be addressed by.

Empty in, empty out. The SDK panics rather than errors on a contradictory worker configuration, so the contradictions are made unrepresentable here instead.

Half in is an error, and used not to be. Dropping silently to unversioned is the worst of the three answers: the operator asked for versioning, the worker starts, nothing says otherwise, and the guarantee they configured is simply absent — a fail-open on the exact posture the interpreter depends on. Somebody who set one flag meant to set both, so the missing half is named and the command stops.

func Register

func Register(w worker.Registry, runtime ...TaskRuntimeConfig)

Register installs the interpreter on a worker.

Every process that runs Flowstate workloads calls this rather than registering by hand. There were four hand-written copies of these five lines before it existed, which is four places to forget the versioning behaviour, and four places to forget that TaskWithPrev still has to be registered for runs that predate scopes. A registration list is a thing to get exactly right once.

The parameter is worker.Registry rather than worker.Worker so that anything that can hold registrations can be passed one — including a test environment wrapping a real worker.

func RegisterWorkflows

func RegisterWorkflows(r WorkflowRegistry)

RegisterWorkflows installs the interpreter workflow, and only the workflow.

The single caller that is not Register is the replay gate. Keeping the options here, in one function both reach, is the same argument Register's own doc makes about its five lines: a registration is a thing to get exactly right once.

One *static* type, and never Temporal's dynamic workflow registration, which answers the same "one handler, many workloads" need by selecting a fallback handler on the workflow type name the caller started. Here the workload arrives as a typed v1.RunState argument instead, which is what gives one type to pin versioning to for the whole fleet, one name the replay corpus can register against (replay_test.go registers through this very function), and one place determinism is enforced rather than one per workload. That is why recordingRegistry's RegisterDynamicWorkflow in versioning_test.go is empty: nothing is registered there on purpose. The trade is that every run's WorkflowType is "Run" — see docs/ARCHITECTURE.md, "One interpreter, not a workflow type per workload", for what carries a workload's own name instead.

func Run

Run is the durable workflow entrypoint that supports Continue-As-New. It executes from the provided state and yields final step outputs when done.

A thin wrapper around [runWorkflow] rather than the whole body itself, so that the one thing every terminal failure needs — being reported as a temporal.ApplicationError whose Type is the run's v1.ErrorKind — happens at a single choke point instead of at each of runWorkflow's several return statements. [classifyRunError] is what does that, and it is careful to leave a Continue-As-New error and a cancellation untouched; see its own comment for why touching either would be wrong.

Registered as the workflow function itself (not runWorkflow), and passed by this same name to Continue-As-New below — both matter, because Temporal resumes a continued run by looking up the registered function by the value passed to NewContinueAsNewErrorWithOptions, and a workflow's own dispatch table always points at the registered name.

func RunAddressFrom

func RunAddressFrom(workflowID, firstRunID, currentRunID string) *v1.RunAddress

RunAddressFrom builds a run's address from the three things Temporal knows about an execution, and choosing between the last two is its whole substance.

Continue-As-New starts a *new* Temporal execution with the same workflow id and a fresh run id, and this engine continues as new on its own schedule — a step budget an author never sees and cannot predict. A workload that handed out `currentRunID` would therefore report one address before it suspended and a different one after, with nothing in the file to explain it. `firstRunID` (Temporal's `FirstRunID`) is preserved along the whole chain of continued executions, so it names the run an author believes they wrote. See [v1.RunAddress.run_id].

The fallback is for the one source that does not offer the first id at all: Temporal's own workflow test environment leaves it unset. Falling back to the current execution is correct there rather than merely convenient, because a run that has never continued as new has one execution and the two ids are the same value.

Exported and taking plain strings so the choice can be tested directly. It is exactly the kind of rule a test environment cannot exercise — it never populates the field the rule is about — and a bound nothing reaches is a bound nothing tests.

func Task

func Task(ctx context.Context, task *v1.Task, identity *v1.WorkloadIdentity, continueOnError bool, stepID string) (*v1.Node_Outputs, error)

Task is a Temporal activity that executes a single task.

The workflow pre-resolves expression inputs to literals before scheduling this activity, which keeps the payload small and avoids carrying growing prior outputs across every step.

identity is the run's own attested v1.WorkloadIdentity (or nil, for a local run or a run that predates the field), threaded in from [executor.dispatch]'s `e.identity` the same way [taskActivities.TaskAuthorized] already receives it — one source, one spelling, for every entry point that can carry identity at all. Added as a parameter rather than read from anywhere ambient, because this activity — unlike TaskInScope — never receives a v1.Scope, so there is nothing else on this call to carry it.

continueOnError is the step's own `continue_on_error:` (#750, v1.StepPolicy.GetContinueOnError), threaded the identical way: read at [executor.dispatch] from the node's policy, which is workflow-side and already in hand there, and carried across the activity boundary as a parameter because that is the only way it reaches [activityError], which runs here on the worker. It only ever categorizes the *Temporal* error this activity returns — see [activityError]'s own doc for why that is a heuristic and not a correctness signal.

stepID is the id of the step this dispatch is for, carried so the task span this activity opens names it — the same fact [taskActivities.TaskAuthorized] has always received, now on this arm too. It is *last* rather than beside the other identity-ish parameters, and that position is load-bearing: see the appended-parameter rule in versioning.go. An activity task scheduled by an older interpreter carries no payload for it and arrives here as the empty string, which v1.StartTaskSpan already treats as "no id known" and omits.

func TaskInScope

func TaskInScope(ctx context.Context, task *v1.Task, scope *v1.Scope, continueOnError bool, stepID string) (*v1.Node_Outputs, error)

TaskInScope executes a task that evaluates expressions itself, against the scope those expressions resolve against.

The scope carries both earlier step outputs and any variables bound by enclosing control flow — a loop's current item, a name a step's own `vars:` block declared. Sending it is what lets a task inside a loop body evaluate an expression naming one of those, since that evaluation happens here on the worker rather than in workflow code.

The `http` task is the one that needs it today: `expect:` and `outputs:` are checked against a response that does not exist until the request has been made, so they cannot be resolved before the activity is scheduled — and they may still name a binding from the loop the step sits in.

continueOnError carries the step's `continue_on_error:` across the activity boundary the same way it does on Task — see that parameter's doc.

stepID is appended for the reason, and under the rule, Task's is.

func TaskWithPrev

func TaskWithPrev(ctx context.Context, task *v1.Task, prev *v1.Workflow_StepOutputs) (*v1.Node_Outputs, error)

TaskWithPrev executes a task that evaluates expressions itself and therefore needs the outputs of earlier steps.

Retained so that a workflow started before scopes existed continues to run; new runs schedule TaskInScope instead — Register's own comment states this is not dead code, only uncalled by anything currently scheduling activities: a run whose history already recorded a `TaskWithPrev` schedule (before TaskInScope existed) is replayed against exactly the arguments that were persisted, so this activity's signature is frozen at what a pre-scope run could have recorded — unlike Task, it cannot gain an identity parameter without breaking every such run still in flight the day this deploys. identity therefore stays nil here, deliberately: no pre-scope run ever carried one to lose, and v1.WorkloadIdentity nil reads as every field empty, exactly as an absent one always has.

The same freeze rules out a continueOnError parameter (#750): a pre-scope run's history never recorded one, so it stays uncategorized here rather than benign — the frozen signature has no way to carry it, and inventing a value this activity was never told would be a guess dressed as a fact.

func TenantInterceptor

func TenantInterceptor(namespace string) interceptor.WorkerInterceptor

TenantInterceptor restricts a worker to one Flowstate namespace.

Per-tenant task queues (TaskQueues) are how a tenant's runs are *addressed* to that tenant's worker fleet. This is what makes getting the addressing wrong an answer rather than a silence: a run that reaches a worker it does not belong to is refused, loudly and terminally, instead of being executed by a process holding another tenant's secrets, egress policy, and plugins.

That is the difference the issue this implements names — fail-closed rather than fail-quiet. A misrouted run failing is a page somebody acts on; a misrouted run *succeeding* is a tenancy breach that leaves no trace at all, because every later request about it is still authorized against the run's own recorded tenant and still answers correctly.

What it guards

The run, at the workflow entry point, which is the boundary that matters: a refused run schedules no activity, resolves no secret, and reaches no plugin. A workflow whose arguments do not carry a v1.RunState at all is refused too — a worker restricted to one tenant cannot tell whose work an unrecognized workflow is, and guessing is the fail-open answer.

Also every activity whose arguments say whose work it is, as a second line for the case a wrong-tenant worker shares a queue with a right-tenant one and steals an activity task from a run the right-tenant worker already accepted. That is Task and the authorized arms, which take an identity parameter, and TaskInScope and WorkflowVars, which carry one inside their v1.Scope.

A scope answers this question whether or not it holds an identity, which [tenantArg] argues at length because it is the thing that has been got wrong in both directions: the default tenant's namespace is the empty string, so a scope naming no identity names the default tenant rather than declining to answer, and a worker restricted to another tenant must refuse it.

TaskWithPrev is the one arm carrying neither shape, which is precisely why it has nothing to check — it predates scopes and exists only to replay histories that name it.

So this is defense in depth and not the boundary; the boundary is the queue plus the run refusal above.

What the refusal costs

Both refusals are non-retryable, so a run that reaches the wrong worker fails rather than being retried until a right one happens to pick it up. Retrying would often work — and would leave the misconfiguration in place, unreported, until the day no correctly-configured worker is polling. A bound nothing reaches is a bound nothing tests, and a misconfiguration nothing reports is a misconfiguration nobody fixes.

What the refusal says

It names the run's own namespace and never the worker's. The run's tenant is something its owner already knows; the worker's tenant is another tenant's name, and writing it into a failure that lands in this run's history would disclose the deployment's tenancy to the wrong party — the same reason FlowstateServer.clientFor names only the caller's own namespace when it refuses. An operator gets the whole picture from the worker's own logs, which are not tenant-readable.

func UseCodec

func UseCodec(cfg payloadcodec.Config)

UseCodec is UseDataConverter spelled in terms of the codec slot, so a caller wiring a worker never has to build the converter itself and never has a chance to pair the codec converter with a plain failure converter.

func UseDataConverter

func UseDataConverter(dc converter.DataConverter)

UseDataConverter tells the interpreter which data converter the worker is built with.

Called once, at worker construction, before Register, and from nowhere else. A deployment that never calls it gets converter.GetDefaultDataConverter, which is what every deployment had before payload codecs existed.

func WorkflowVars

func WorkflowVars(ctx context.Context, declared *v1.Scope) (*v1.Scope, error)

WorkflowVars is a Temporal activity that evaluates a workflow's `vars:` block.

Why an activity, for expressions with no side effects at all

The reason usually given is that evaluating CEL in workflow code is not deterministic in the sense replay needs: a language profile pins which *functions* exist and not how cel-go implements them, so an upstream bug fix changes a result under an unchanged profile. That is true, and it is not the reason — because the executor evaluates CEL in workflow code all over the place and always has. A step's condition, a loop's `items:`, a step's own `vars:` block, and the inputs of every task that does not declare NeedsPrevOutputs are all resolved inline, in workflow code, and their results reach history as the arguments of the activities they schedule.

So that exposure is accepted rather than avoided, mitigated where Worker Versioning pins the interpreter and named here so nobody concludes from this activity that it was solved. Routing each of those through an activity would be a round trip per condition.

What makes the workflow's `vars:` different is Continue-As-New, which versioning does not reach. A later segment *replays nothing* — it starts from RunState rather than from history, which is exactly what makes suspending cheap — so a `vars:` block evaluated inline would be evaluated again at the top of every segment, against whatever cel-go that worker has. A value that changed halfway through a run is a worse failure than a replay mismatch, because nothing detects it. Evaluated once in an activity and carried in RunState, it cannot.

Not the same reason TaskInScope exists, which is easy to assume from the two sitting side by side. That activity carries a *scope* to the worker because the expressions it evaluates name things the workflow does not have — a loop's binding, and the `response.*` of a request that has not been made yet — and it is reached only by tasks declaring NeedsPrevOutputs, which today is `http` and plugins asking for a scope. See its own doc.

docs/DSL.md holds open the faster path — workflow-side evaluation where Worker Versioning pins the interpreter — but that would not help here: versioning pins the interpreter within a run's *history*, and the next segment has none.

Takes and returns a v1.Scope rather than the specification: the vars go in unevaluated and come back evaluated, alongside the profile they are evaluated against, and nothing else about the workflow is needed or shipped.

Types

type ErrRunFailed

type ErrRunFailed struct {
	Message string

	// Recorded is the driver-independent text this failure records as the step's
	// `error` output when `continue_on_error` tolerates it — rendered by
	// [v1.StepErrorText], and deliberately not Message.
	//
	// Message formats the whole cause, Temporal's envelope included, for the
	// run-level failure a person reads. Recorded is the value an author's
	// expression compares, so it has to be the same sentence the local driver
	// records for the same failure.
	//
	// It travels as a field rather than as a wrapped cause because this type has
	// no general wrapped cause: Temporal's failure converter walks the unwrap
	// chain into the failure it persists, and what this deliberately flattens
	// must stay flattened. The sole exception is a ScheduleToClose expiry during
	// retry backoff, whose last application failure is retained below as the
	// structured evidence for the budget judgement (#1163).
	Recorded string

	// Kind classifies the failure the same way [v1.ClassifyError] would,
	// carried alongside Message and Recorded for the same reason both of those
	// are: whatever wrapping this driver adds between where the failure
	// happened and where the run finally reports it must not be what a caller
	// reads back. See [recordedStepKind], which computes it, and
	// [classifyRunError], which is what puts it where a client can read it —
	// this field only carries it there.
	Kind v1.ErrorKind
	// contains filtered or unexported fields
}

func (*ErrRunFailed) Error

func (e *ErrRunFailed) Error() string

func (*ErrRunFailed) Unwrap

func (e *ErrRunFailed) Unwrap() error

type TaskQueues

type TaskQueues struct {
	// Prefix names the family of per-tenant queues. Empty routes every run to
	// [RunTaskQueueName], the zero-configuration path.
	//
	// It is checked against [auth.ValidateNamespace]'s grammar, because that is
	// the grammar the forgery argument above rests on — not because a prefix is
	// a namespace.
	Prefix string
}

TaskQueues decides which Temporal task queue a run is submitted to.

The zero value is the whole of today's behavior: every run, of every tenant, goes to RunTaskQueueName. That is not a compatibility shim to be removed — it is the single-tenant deployment, where there is nothing to route between, and it must stay byte-identical to what a deployment that never heard of this type already gets.

Setting Prefix turns each tenant's runs onto a queue of its own, which is what makes a per-tenant worker fleet addressable: a worker started with `flow worker --tenant acme --task-queue-prefix <same prefix>` polls exactly the queue this composes for acme, and (see TenantInterceptor) refuses anything else that reaches it.

Why a composed name cannot be forged

The failure to avoid is the one CLAUDE.md records for the env secrets provider: namespace "team" with secret "A_API_KEY" and the default tenant with "TEAM_A_API_KEY" both resolved one variable, because every character legal in a prefix was also legal in a name, so the boundary between them was a convention rather than a fact. No separator fixes that.

This is fixed the way auth.SubjectFor's `_default` is, and it is a structural argument rather than a careful one:

  • A namespace is auth.ValidateNamespace's grammar — lowercase letters, digits, and a dash that is never first. It cannot contain "_".
  • A prefix is checked against the *same* grammar by TaskQueues.Validate, which runs when configuration loads. It cannot contain "_" either.
  • The composed name is `prefix + "_" + tenant`, and the tenant component is the namespace itself, or [defaultTenantComponent] for the empty one.

So the first "_" in a composed name is always the separator, at exactly len(prefix). If two (prefix, namespace) pairs composed the same string, the first "_" would sit at both len(prefix₁) and len(prefix₂), so the prefixes have the same length, so they are equal, so the namespaces are too. There is no pair of distinct inputs left to collide, and no namespace an operator can name that spells another tenant's queue — including the default tenant's, whose component starts with the one character the namespace grammar refuses.

func (TaskQueues) Enabled

func (q TaskQueues) Enabled() bool

Enabled reports whether this deployment routes tenants to their own queues.

func (TaskQueues) For

func (q TaskQueues) For(namespace string) (string, error)

For returns the task queue a run belonging to the given Flowstate namespace is submitted to.

Unconfigured, it answers RunTaskQueueName for every namespace and never errors — including for a namespace auth.ValidateNamespace would refuse. That is deliberate and is the byte-identical default the issue asks for: a run whose recorded identity predates that grammar (see [FlowstateServer.identityFor]) starts today, and must keep starting.

Configured, it fails closed. A namespace outside the grammar cannot be composed into a queue name whose boundary is trustworthy, and the answer to "which queue does this un-namespaceable tenant use" is not "the one everybody else uses" — that is the fallback [temporalclient.Pool.For] already refuses to make, for the same reason.

func (TaskQueues) Validate

func (q TaskQueues) Validate() error

Validate reports whether the configuration is usable.

Called when configuration loads — at `flow server` and `flow worker` startup — rather than when a run is submitted, so a prefix that cannot compose a legal queue name stops the process instead of failing every submission after it.

type TaskRuntimeConfig

type TaskRuntimeConfig struct {
	// contains filtered or unexported fields
}

TaskRuntimeConfig is the immutable configuration owned by one worker: the sensitive capabilities its tasks run with, and the plugin inventory its runs are admitted against. It is passed to activity registration rather than stored globally, so two workers embedded in one process cannot overwrite each other's tenant or federation configuration — or each other's answer to "which plugins is this worker holding".

The inventory is not a capability, and naming it here widens what this type is about. That is the cost being paid, deliberately: the alternative is a second per-worker vehicle beside this one, and the whole defect [WithPluginCatalog] closes (#777) is that the catalog had a second vehicle — a process global — which the last worker to be constructed won. One thing carrying everything one worker owns cannot disagree with itself about which worker it belongs to.

func NewTaskRuntimeConfig

func NewTaskRuntimeConfig(store *secrets.Store, policy *auth.SecretPolicy, broker *auth.Broker) (TaskRuntimeConfig, error)

NewTaskRuntimeConfig validates and assembles worker task capabilities.

func (TaskRuntimeConfig) WithPluginCatalog

func (c TaskRuntimeConfig) WithPluginCatalog(catalog *v1.PluginCatalog) TaskRuntimeConfig

WithPluginCatalog returns a copy carrying the plugins this worker actually has.

Called with what the worker's plugin host launched — see cmd/flow's startPlugins — and the result passed to Register, before the worker polls. A worker registered without one has no plugins, which is the truthful answer for a stock worker and the fail-closed one for a worker whose operator forgot: every run pinned to a plugin is refused by the admission check in plugins.go rather than executed by a worker that has none of it.

A copy rather than a mutation because the zero value has to keep meaning "no plugins" for every worker that never says otherwise, and a builder that mutated a shared value would be the process global again wearing a method's clothes.

It is separate from NewTaskRuntimeConfig rather than a fourth parameter to it because the two answer to different owners: the store, policy and broker are a deployment's grant of authority to this worker's tasks and are checked against each other, while the catalog is an observation of what this process launched and can only be wrong by being somebody else's.

type WorkflowRegistry

type WorkflowRegistry interface {
	RegisterWorkflowWithOptions(w any, options workflow.RegisterOptions)
}

WorkflowRegistry is the workflow half of worker.Registry.

It exists because the other thing that has to hold this package's workflow registration is not a worker at all: worker.WorkflowReplayer, which replays a recorded history against current code and therefore registers workflows and nothing else — an activity is never executed during a replay, only named by the history. Narrowing the parameter is what lets the replay corpus in replay_test.go register Run the way a real worker does rather than by hand, which matters more here than it looks: replaying against different registration options than production uses would make the gate answer a question nobody asked.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL