Documentation
¶
Overview ¶
Package flowgraph is a lazy, demand-driven, type-enforced module graph.
Caution: this library is fully AI-generated ¶
The implementation, tests, examples and documentation were all written by an AI agent.
The test suite in the repository exercises every rule described below, but the only real-world validation is against production workloads specific to Heru, Inc. Anything those workloads do not exercise — other concurrency shapes, other scales, other cancellation and failure patterns — is unproven, and the concurrency and lifetime semantics are exactly where that gap is most likely to hurt. Use it with caution: read the code, and treat the rules below as claims to verify against your own workload rather than as guarantees.
Overview ¶
The whole system is one idea: a module is a lazy, memoized, shareable async computation that produces exactly one typed value. Nothing runs until its output is demanded; demand propagates backward from whoever ultimately needs the answer; and a value computed once is shared by every later consumer.
Composition is recursion: a node's body may construct and wire its own child nodes, so an "orchestrator" is just a node, and the "controller" is just the root node you call Get on. There is no central registry and no interface{} on any value path — dependencies flow as typed Node handles. The single type-erased seam, Prefetchable, deliberately exposes no value.
The four rules ¶
Laziness. A node whose Get or Prefetch is never called never runs. A module that is not needed under the current inputs costs nothing.
Memoization. A node's body runs at most once per node instance. Concurrent demanders join the one in-flight execution; later demanders get the stored result immediately.
Owner-context execution. A node body runs under its owning Scope's context — never a caller's. So a consumer that walks away (cancellation, early failure) abandons only its own wait: the node keeps running and memoizes for everyone else. Cancel a scope to cancel the nodes that scope owns. Lifetime cascades by ownership, never by consumption.
Hierarchical sharing with outermost preference. Shared dependencies are registered into a scope chain by their type (Provide) and looked up by walking upward (Resolve). The outermost provider wins, so the widest set of consumers shares one instance. Override caps that walk for one subtree when it genuinely needs its own binding.
Shape of the API ¶
Go forbids generic methods, so Compute, Provide, Resolve and every combinator are free functions taking a *Scope. That is forced by the language, not a style choice; Node[T].Get is a method because Node[T] is a generic type rather than a generic method.
Build nodes with Compute, derive them with Then, the Zip2 to Zip5 family and All, fan out with Map, MapCollect and Expand, and recover with Fallback and Race. PrefetchAll starts a set of differently-typed nodes at once. The core is deliberately tiny; the combinators exist so applications never need to touch its internals.
Parameterised graphs ¶
Sharing a value means sharing the node, which is automatic for a node built once and passed around, and not automatic at all once nodes come from a factory taking an argument — two calls build two nodes, and two nodes run twice. Memo with Memoize keys nodes so that one key means one node and one execution.
Seeing what happened ¶
A lazy graph hides what ran, what was skipped, how long each node took and what overlapped. Label names a node, both in trace events and in the errors it returns, so a failure carries its demand path. WithObserver installs a scope-level hook receiving an Event per node as it is demanded, starts and finishes; child scopes inherit it. Both are opt-in and free when unused.
Index ¶
- func Gather[B any](ctx context.Context, nodes []Node[B]) ([]B, error)
- func GetWithTimeout[T any](ctx context.Context, n Node[T], d time.Duration) (T, error)
- func Override[T any](s *Scope, n Node[T])
- func PrefetchAll(nodes ...Prefetchable)
- func Provide[T any](s *Scope, n Node[T])
- type Event
- type EventKind
- type Memo
- type Node
- func All[T any](s *Scope, nodes []Node[T], opts ...NodeOption) Node[[]T]
- func Compute[Out any](s *Scope, run func(ctx context.Context) (Out, error), opts ...NodeOption) Node[Out]
- func Const[T any](s *Scope, v T, opts ...NodeOption) Node[T]
- func Expand[A, B any](s *Scope, list Node[[]A], f func(ctx context.Context, item A) (B, error), ...) Node[[]Node[B]]
- func Fallback[T any](s *Scope, primary, secondary Node[T], opts ...NodeOption) Node[T]
- func Map[A, B any](s *Scope, list Node[[]A], maxParallel int, ...) Node[[]B]
- func MapCollect[A, B any](s *Scope, list Node[[]A], maxParallel int, ...) Node[[]Result[B]]
- func Memoize[K comparable, T any](m *Memo, key K, build func() Node[T]) Node[T]
- func MustResolve[T any](s *Scope) Node[T]
- func Race[T any](s *Scope, candidates []Node[T], opts ...NodeOption) Node[T]
- func Resolve[T any](s *Scope) (Node[T], bool)
- func Then[A, B any](s *Scope, a Node[A], f func(ctx context.Context, a A) (B, error), ...) Node[B]
- func Zip2[A, B, C any](s *Scope, a Node[A], b Node[B], ...) Node[C]
- func Zip3[A, B, C, D any](s *Scope, a Node[A], b Node[B], c Node[C], ...) Node[D]
- func Zip4[A, B, C, D, E any](s *Scope, a Node[A], b Node[B], c Node[C], d Node[D], ...) Node[E]
- func Zip5[A, B, C, D, E, F any](s *Scope, a Node[A], b Node[B], c Node[C], d Node[D], e Node[E], ...) Node[F]
- type NodeOption
- type Prefetchable
- type Result
- type Scope
- type ScopeOption
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Gather ¶
Gather forces every node in a slice concurrently (Prefetch all, then Get each) and collects the results in order. The natural terminal step after Expand when a consumer does want all items.
It returns the first error in slice order, having already started every node; the nodes it stops waiting on keep running under their owners' contexts and stay available to later consumers.
func GetWithTimeout ¶
GetWithTimeout bounds how long THIS CALLER waits. On timeout the node keeps running (owner lifetime) and its result stays available to later callers — a later Get can pick up the memoized value.
func Override ¶
Override registers n as a provider of T that CAPS upward resolution. This scope and everything below it resolve to n regardless of any outer Provide. Use it when a subtree legitimately needs a different T than its ancestors.
An Override is downward-only: ancestors and sibling subtrees never see it.
func PrefetchAll ¶ added in v0.2.0
func PrefetchAll(nodes ...Prefetchable)
PrefetchAll starts every node given, so a set of differently-typed dependencies begins working concurrently before anything is collected. It is the idiomatic first half of "start everything, then gather each result":
flowgraph.PrefetchAll(orders, rates, inventory) o, err := orders.Get(ctx) // all three are already in flight
Prefetch is idempotent, so calling this repeatedly, or alongside a later Get, is harmless.
Example ¶
PrefetchAll starts a set of differently-typed nodes at once, so they overlap instead of running one after another.
package main
import (
"context"
"fmt"
"time"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
name := flowgraph.Compute(root, func(ctx context.Context) (string, error) {
time.Sleep(30 * time.Millisecond)
return "widget", nil
})
price := flowgraph.Compute(root, func(ctx context.Context) (int, error) {
time.Sleep(30 * time.Millisecond)
return 250, nil
})
stock := flowgraph.Compute(root, func(ctx context.Context) (bool, error) {
time.Sleep(30 * time.Millisecond)
return true, nil
})
// Three output types: no common value, but they all satisfy Prefetchable.
started := time.Now()
flowgraph.PrefetchAll(name, price, stock)
n, _ := name.Get(ctx)
p, _ := price.Get(ctx)
s, _ := stock.Get(ctx)
fmt.Printf("%s: %d cents, in stock: %v\n", n, p, s)
fmt.Println("overlapped:", time.Since(started) < 60*time.Millisecond)
}
Output: widget: 250 cents, in stock: true overlapped: true
func Provide ¶
Provide registers n as a shared provider of T. It participates in outermost-preference: when several scopes Provide T, Resolve picks the one closest to the root, so the widest set of consumers shares one instance.
Providing T twice in the same scope replaces the earlier registration.
Example ¶
ExampleProvide shares one node with consumers at unknown depths. Resolution walks upward and the outermost provider wins, so the widest set of consumers shares a single instance; Override caps that walk for one subtree.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
// A distinct named type is the sharing key — never a bare builtin.
type Region string
root := flowgraph.NewScope(ctx, nil)
service := root.Child(ctx)
tenant := service.Child(ctx)
flowgraph.Provide(root, flowgraph.Const(root, Region("us-east-1")))
// A plain Provide deeper down does not shadow the outer one: sharing beats
// lexical scoping by default.
flowgraph.Provide(tenant, flowgraph.Const(tenant, Region("eu-west-1")))
got, _ := flowgraph.MustResolve[Region](tenant).Get(ctx)
fmt.Println("tenant resolves:", got)
// An Override does shadow it, for this scope and its descendants only.
flowgraph.Override(tenant, flowgraph.Const(tenant, Region("eu-west-1")))
got, _ = flowgraph.MustResolve[Region](tenant).Get(ctx)
fmt.Println("after override: ", got)
sibling := service.Child(ctx)
got, _ = flowgraph.MustResolve[Region](sibling).Get(ctx)
fmt.Println("sibling still: ", got)
}
Output: tenant resolves: us-east-1 after override: eu-west-1 sibling still: us-east-1
Types ¶
type Event ¶ added in v0.2.0
type Event struct {
// Kind is which moment this reports.
Kind EventKind
// ID identifies the node, and is stable across that node's events. It exists
// so unlabeled nodes can still be told apart; prefer [Label] for anything you
// intend to read. IDs are assigned per node built under an observed scope, in
// construction order, and are not meaningful across scopes.
ID uint64
// Label is the node's [Label], or "" if it was not labeled.
Label string
// Duration is how long the body ran. Set on EventFinished only.
Duration time.Duration
// Err is the error the node memoized, already wrapped with the node's label if
// it has one. Set on EventFinished only, and nil on success.
Err error
}
Event is one observation of a node's execution, delivered to the observer installed with WithObserver.
Together the events answer the questions a lazy graph otherwise hides: what actually ran, what was skipped, how long each node took, what overlapped, and how many consumers shared each value.
type EventKind ¶ added in v0.2.0
type EventKind int
EventKind says which moment in a node's life an Event reports.
const ( // EventDemanded reports a Get or Prefetch call. It fires on EVERY call, // including ones that hit the memoized value and so start no work — which is // what makes it useful: a node with many Demanded events and one Started event // is the sharing this package rests on, made visible. // // It counts CALLS, not distinct consumers. A join is a single consumer but // demands each of its sides twice, because [Zip2] and its siblings prefetch // every side and then get each one. EventDemanded EventKind = iota // EventStarted reports the body beginning to run. It fires at most once per // node, and only for nodes that are actually demanded — a node with Demanded // events but no Started event was answered from memory, and a node with no // events at all was never needed. EventStarted // EventFinished reports the body returning, with [Event.Duration] and the // [Event.Err] the node will hand to its consumers. It fires at most once per // node, and always before the node's waiters unblock. EventFinished )
type Memo ¶ added in v0.2.0
type Memo struct {
// contains filtered or unexported fields
}
Memo is a keyed cache of nodes: ask for the same key twice and get back the SAME node, so the memoization everything else here rests on keeps applying across consumers.
It is what a parameterised graph cannot do without. Compute builds a FRESH node on every call, and each node memoizes independently — so a factory like
func series(s *flowgraph.Scope, id string) flowgraph.Node[Series]
called twice with the same id yields two nodes and two executions, and every consumer deriving from "the" series for that id gets a private copy of the work. Routing those calls through a Memo restores the invariant: one key, one node, one execution, shared by every consumer.
Keys are scoped by output type as well as by value, so key "x" for a Node[A] and key "x" for a Node[B] are separate entries that cannot collide.
The zero value is ready to use, and a Memo is safe for concurrent use.
A Memo does not own the nodes it caches. Lifetime still belongs to the scope each node was built under, so a Memo outliving that scope will keep handing out nodes whose owner context is already cancelled. Give a Memo the same lifetime as the scope whose nodes it caches — usually by building both in the same place.
type Node ¶
type Node[Out any] interface { // Get forces the node (if not already started) and blocks until the result // is ready or the CALLER's context is cancelled. Crucially, the node body // runs under its OWNER's context, not the caller's — so a consumer that // walks away (early failure, lost interest) never cancels the node. The // node keeps running and memoizes its result for everyone else. That single // split is what delivers isolation + cancel-immune shared providers. // // A cancelled caller gets ctx.Err(). The one exception is a node that has // already finished: its stored result is returned even to a cancelled // caller, because there is no longer any work to abandon. Get(ctx context.Context) (Out, error) // Prefetch starts the node without blocking. This is request-mode (a): // "I'll need this, start working on it, I'll check later." Get is mode (b): // "I need this now." Both are idempotent; the body runs at most once. Prefetch() }
Node is a lazy, memoized, shareable computation producing one Out.
A Node is a handle, not a value: holding one costs nothing and starts nothing. The body runs at most once per Node instance, however many consumers demand it and however often.
func All ¶
func All[T any](s *Scope, nodes []Node[T], opts ...NodeOption) Node[[]T]
All joins any number of same-typed nodes: prefetch all, gather in order. See Gather for the error semantics.
func Compute ¶
func Compute[Out any](s *Scope, run func(ctx context.Context) (Out, error), opts ...NodeOption) Node[Out]
Compute builds a node owned by scope s. The node's body runs under s's context, so the node lives exactly as long as its owning scope — never as long as a consumer. Tear down a scope to cancel everything it owns.
Building a node is pure bookkeeping: run is not called until something demands the node. A node body may itself call Compute to build child nodes, which is how the graph grows at runtime (see Expand).
A panic inside run is not recovered. Because the body runs on its own goroutine, a panic crashes the program exactly as any unrecovered goroutine panic does, and the demanding goroutine's stack will not appear in the trace. Return errors from node bodies rather than panicking.
Example ¶
ExampleCompute shows the two properties every node has: it does not run until something demands it, and it runs at most once however many consumers ask.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
runs := 0
price := flowgraph.Compute(root, func(ctx context.Context) (int, error) {
runs++
fmt.Println("expensive lookup running")
return 21, nil
})
fmt.Println("graph built, runs so far:", runs)
a, _ := price.Get(ctx)
b, _ := price.Get(ctx) // second consumer: memoized, no second run
fmt.Printf("values: %d %d after %d run\n", a, b, runs)
}
Output: graph built, runs so far: 0 expensive lookup running values: 21 21 after 1 run
func Const ¶
func Const[T any](s *Scope, v T, opts ...NodeOption) Node[T]
Const wraps an already-known value as a node, so plain values can be handed to anything expecting a Node[T] (inputs, defaults, tests).
func Expand ¶
func Expand[A, B any]( s *Scope, list Node[[]A], f func(ctx context.Context, item A) (B, error), opts ...NodeOption, ) Node[[]Node[B]]
Expand is the NODE-SHAPED fan-out: dynamic graph expansion. It forces the list node and mints one lazy child Node[B] per item, returning the slice of handles WITHOUT forcing any of them.
Use this when per-item results need node properties: several downstream consumers demand individual items independently (each item computes once, memoized, shared); or some items may never be demanded at all (per-item laziness). The item nodes are owned by scope s, so they inherit its lifetime and are immune to any single consumer's cancellation — same rules as every other node.
Note the type is Node[[]Node[B]]: a node whose VALUE is nodes. That is legal here precisely because composition is recursion; the graph grows at runtime to match data only knowable at runtime. Gather is the usual terminal step when a consumer does end up wanting every item.
Example ¶
ExampleExpand promotes list items to nodes. Minting the handles runs nothing; each item then computes at most once, no matter how many consumers want it, and items nobody demands never run at all.
package main
import (
"context"
"fmt"
"strings"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
runs := map[string]int{}
pages := flowgraph.Const(root, []string{"cover", "body", "appendix"})
rendered := flowgraph.Expand(root, pages, func(ctx context.Context, page string) (string, error) {
runs[page]++
return strings.ToUpper(page), nil
})
handles, _ := rendered.Get(ctx)
fmt.Println("handles minted:", len(handles), "renders so far:", len(runs))
// Two consumers both want the cover; nobody wants the appendix.
first, _ := handles[0].Get(ctx)
again, _ := handles[0].Get(ctx)
body, _ := handles[1].Get(ctx)
fmt.Println(first, again, body)
fmt.Println("cover renders:", runs["cover"], "appendix renders:", runs["appendix"])
}
Output: handles minted: 3 renders so far: 0 COVER COVER BODY cover renders: 1 appendix renders: 0
func Fallback ¶
func Fallback[T any](s *Scope, primary, secondary Node[T], opts ...NodeOption) Node[T]
Fallback forces primary; on error it forces secondary instead. The primary's error is wrapped into the secondary's error if BOTH fail. Laziness holds: secondary never runs if primary succeeds — which is what makes a chain of Fallbacks the natural spelling of a config cascade.
Example ¶
ExampleFallback is the layered-config cascade: laziness means each tier is only consulted if the tier above it failed.
package main
import (
"context"
"errors"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
fromEnv := flowgraph.Compute(root, func(ctx context.Context) (string, error) {
fmt.Println("checking env")
return "", errors.New("FLOWGRAPH_DSN unset")
})
fromFile := flowgraph.Compute(root, func(ctx context.Context) (string, error) {
fmt.Println("reading config file")
return "postgres://from-file", nil
})
defaults := flowgraph.Compute(root, func(ctx context.Context) (string, error) {
fmt.Println("falling back to defaults") // never printed: the file tier won
return "postgres://localhost", nil
})
dsn := flowgraph.Fallback(root, fromEnv, flowgraph.Fallback(root, fromFile, defaults))
v, _ := dsn.Get(ctx)
fmt.Println("dsn:", v)
}
Output: checking env reading config file dsn: postgres://from-file
func Map ¶
func Map[A, B any]( s *Scope, list Node[[]A], maxParallel int, f func(ctx context.Context, item A) (B, error), opts ...NodeOption, ) Node[[]B]
Map is the SIMPLE fan-out: a node that forces the list node, then transforms every item in parallel (bounded by maxParallel; <=0 means unbounded) and memoizes the whole []B as one value. Output order matches input order.
Use this when the per-item work is a private implementation detail of the consumer: no other node will ever demand a single item's result, so the items don't need to be nodes. Structured concurrency inside a node body is ordinary Go — not a violation of the architecture.
Error semantics are fail-fast WITHIN the batch (the first error cancels the sibling items via the group context, and Map returns that error) but isolated OUTSIDE it: the group context derives from this node's own owner context, so a batch failure never propagates to other nodes, and a consumer walking away never cancels the batch.
Example ¶
ExampleMap fans out over a list whose length is only known at runtime, with a concurrency cap. Output order always matches input order.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
docs := flowgraph.Const(root, []string{"alpha", "bravo", "charlie", "delta"})
lengths := flowgraph.Map(root, docs, 2, func(ctx context.Context, d string) (int, error) {
return len(d), nil
})
out, err := lengths.Get(ctx)
fmt.Println(out, err)
}
Output: [5 5 7 5] <nil>
func MapCollect ¶
func MapCollect[A, B any]( s *Scope, list Node[[]A], maxParallel int, f func(ctx context.Context, item A) (B, error), opts ...NodeOption, ) Node[[]Result[B]]
MapCollect is Map's collect-all-errors sibling: every item is processed (no intra-batch fail-fast), and each item's success or failure is reported in order. Use when partial progress matters more than stopping early.
maxParallel bounds concurrency exactly as in Map (<=0 means unbounded). The node itself fails only if the list node fails — an item error is data, carried in that item's Result.
Example ¶
ExampleMapCollect keeps going after a bad item and reports each outcome, where Map would fail the whole batch.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
ids := flowgraph.Const(root, []int{1, -2, 3})
results := flowgraph.MapCollect(root, ids, 3, func(ctx context.Context, id int) (string, error) {
if id < 0 {
return "", fmt.Errorf("id %d: negative", id)
}
return fmt.Sprintf("record-%d", id), nil
})
out, err := results.Get(ctx) // the batch itself succeeds
fmt.Println("batch error:", err)
for i, r := range out {
if r.Err != nil {
fmt.Printf("item %d failed: %v\n", i, r.Err)
continue
}
fmt.Printf("item %d ok: %s\n", i, r.Val)
}
}
Output: batch error: <nil> item 0 ok: record-1 item 1 failed: id -2: negative item 2 ok: record-3
func Memoize ¶ added in v0.2.0
func Memoize[K comparable, T any](m *Memo, key K, build func() Node[T]) Node[T]
Memoize returns the node cached under key, calling build to create it the first time. Go forbids generic methods, so this is a free function over the Memo exactly as Compute is over a Scope.
node := flowgraph.Memoize(memo, id, func() flowgraph.Node[Series] {
return flowgraph.Compute(s, func(ctx context.Context) (Series, error) { … })
})
Two properties of how it does that are worth stating, because both are easy to get wrong by hand and neither is obvious.
build runs with the cache's lock RELEASED. Node factories nest — a derived value's factory calls the factory for what it derives from — and Go's mutexes are not reentrant, so holding the lock across build would deadlock the first time one memoized factory called another.
Because the lock is released, two goroutines can build the same key at once. Only one node is kept, and BOTH callers get that one, so the identity guarantee holds no matter who won. That is safe precisely because building a node runs nothing: the loser's node is an unforced handle, and dropping it costs nothing.
The consequence for callers is the one rule to remember: build may run more than once for a key, so it must do nothing but construct nodes. Anything with a side effect belongs in the node's body, which runs at most once.
Example ¶
Memoize keys nodes so that asking for the same derived value twice returns the same node — and therefore runs it once.
package main
import (
"context"
"fmt"
"strings"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
memo := flowgraph.NewMemo()
var loads int
// A parameterised factory. Without the Memo, each call would build a separate
// node, and each node would load again.
channel := func(id string) flowgraph.Node[string] {
return flowgraph.Memoize(memo, id, func() flowgraph.Node[string] {
return flowgraph.Compute(root, func(ctx context.Context) (string, error) {
loads++
return strings.ToUpper(id), nil
})
})
}
first := channel("alpha")
second := channel("alpha") // same key: the identical node
other := channel("beta")
fmt.Println("same node for the same key:", first == second)
for _, n := range []flowgraph.Node[string]{first, second, other} {
v, _ := n.Get(ctx)
fmt.Println(v)
}
fmt.Println("loads:", loads)
}
Output: same node for the same key: true ALPHA ALPHA BETA loads: 2
func MustResolve ¶
MustResolve is the wiring-phase form: it panics when T has no provider. Do all dependency resolution at graph construction, before any Get, so an unsatisfied dependency fails at build time rather than deep inside a run. This is the wire-time safety floor that stands in for the compile-time checking we trade away for automatic resolution.
func Race ¶
func Race[T any](s *Scope, candidates []Node[T], opts ...NodeOption) Node[T]
Race forces every candidate concurrently and resolves to the FIRST success. Consistent with the isolation rule, losers are NOT cancelled — they keep running under their owners' contexts and memoize for any other consumer. Race only stops WAITING on them. If all candidates fail, the errors are joined (see errors.Join).
func Resolve ¶
Resolve implements outermost-with-local-override. Walking from s up to the root: each provider found overwrites the choice, so the highest (outermost) normal provider wins by default. An Override breaks the walk at its scope, so the most-local override caps the search.
The returned node is not forced. Resolve reports whether a provider exists; use MustResolve when a missing provider is a wiring bug.
func Then ¶
func Then[A, B any](s *Scope, a Node[A], f func(ctx context.Context, a A) (B, error), opts ...NodeOption) Node[B]
Then derives a new node from one upstream node: force a, transform. The building block for linear pipelines.
Example ¶
ExampleThen builds a linear pipeline. Each stage forces the one before it, so demanding the last stage pulls the whole chain — and nothing more.
package main
import (
"context"
"fmt"
"strings"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
raw := flowgraph.Const(root, " Widget, Gadget , Doohickey ")
fields := flowgraph.Then(root, raw, func(ctx context.Context, s string) ([]string, error) {
out := strings.Split(s, ",")
for i := range out {
out[i] = strings.TrimSpace(out[i])
}
return out, nil
})
count := flowgraph.Then(root, fields, func(ctx context.Context, f []string) (int, error) {
return len(f), nil
})
names, _ := fields.Get(ctx)
n, _ := count.Get(ctx) // reuses the memoized fields value
fmt.Println(names, n)
}
Output: [Widget Gadget Doohickey] 3
func Zip2 ¶
func Zip2[A, B, C any](s *Scope, a Node[A], b Node[B], f func(ctx context.Context, a A, b B) (C, error), opts ...NodeOption) Node[C]
Zip2 joins two nodes: both are prefetched (so they run concurrently), then combined. If a side fails, its error propagates and f is not called; the other side still completes under its own owner's context and memoizes.
Example ¶
ExampleZip2 joins two independent nodes. Both are prefetched, so they run concurrently rather than one after the other.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
subtotal := flowgraph.Const(root, 4000)
taxRate := flowgraph.Const(root, 0.07)
total := flowgraph.Zip2(root, subtotal, taxRate,
func(ctx context.Context, cents int, rate float64) (int, error) {
return cents + int(float64(cents)*rate), nil
})
v, _ := total.Get(ctx)
fmt.Println("total cents:", v)
}
Output: total cents: 4280
func Zip3 ¶
func Zip3[A, B, C, D any](s *Scope, a Node[A], b Node[B], c Node[C], f func(ctx context.Context, a A, b B, c C) (D, error), opts ...NodeOption) Node[D]
Zip3 joins three nodes; same semantics as Zip2.
func Zip4 ¶ added in v0.2.0
func Zip4[A, B, C, D, E any](s *Scope, a Node[A], b Node[B], c Node[C], d Node[D], f func(ctx context.Context, a A, b B, c C, d D) (E, error), opts ...NodeOption) Node[E]
Zip4 joins four nodes; same semantics as Zip2.
Example ¶
Zip4 joins four nodes, prefetching every side so they run concurrently.
package main
import (
"context"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
a := flowgraph.Const(root, 1)
b := flowgraph.Const(root, 2)
c := flowgraph.Const(root, 3)
d := flowgraph.Const(root, 4)
sum := flowgraph.Zip4(root, a, b, c, d,
func(ctx context.Context, w, x, y, z int) (int, error) { return w + x + y + z, nil })
v, err := sum.Get(ctx)
fmt.Println(v, err)
}
Output: 10 <nil>
func Zip5 ¶ added in v0.2.0
func Zip5[A, B, C, D, E, F any](s *Scope, a Node[A], b Node[B], c Node[C], d Node[D], e Node[E], f func(ctx context.Context, a A, b B, c C, d D, e E) (F, error), opts ...NodeOption) Node[F]
Zip5 joins five nodes; same semantics as Zip2.
Five is where this family stops. Past that, either group the inputs into a struct and join the structs, or write the join as a Compute body — but if you do, open it with PrefetchAll over every input. Forgetting that is the one failure mode here that nothing reports: the joins still produce correct values, just strictly one after another instead of concurrently.
type NodeOption ¶ added in v0.2.0
type NodeOption func(*nodeConfig)
NodeOption configures a node as it is built. Every function in this package that constructs a node — Compute and all the combinators — accepts these, so a node can be annotated wherever it is created.
func Label ¶ added in v0.2.0
func Label(name string) NodeOption
Label names a node, which does two things.
It gives the node provenance in errors: when a labeled node's body returns a non-nil error, that error is wrapped as "label: original". Wrapping is with %w, so errors.Is and errors.As see straight through it. Because a derived node's body forces its upstreams, labeling a chain yields the demand path for free — a failure three levels down reads as "report: totals: orders: no such file". Unlabeled nodes return their body's error exactly as-is, so this costs nothing until you ask for it.
It also names the node in trace events, which is what makes an observer's output readable. See WithObserver.
Example ¶
Label gives a node provenance in its errors. Because a derived node's body forces its upstreams, labeling a chain yields the whole demand path.
package main
import (
"context"
"errors"
"fmt"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
root := flowgraph.NewScope(ctx, nil)
missing := errors.New("no such file")
raw := flowgraph.Compute(root, func(ctx context.Context) ([]byte, error) {
return nil, missing
}, flowgraph.Label("read(orders.json)"))
orders := flowgraph.Then(root, raw, func(ctx context.Context, b []byte) (int, error) {
return len(b), nil
}, flowgraph.Label("parse"))
report := flowgraph.Then(root, orders, func(ctx context.Context, n int) (string, error) {
return fmt.Sprint(n), nil
}, flowgraph.Label("report"))
_, err := report.Get(ctx)
fmt.Println(err)
fmt.Println("errors.Is sees through the labels:", errors.Is(err, missing))
}
Output: report: parse: read(orders.json): no such file errors.Is sees through the labels: true
type Prefetchable ¶
type Prefetchable interface{ Prefetch() }
Prefetchable is the non-generic view of a node, used to kick off a set of differently-typed dependencies (e.g. an orchestrator holding []Prefetchable) without ever surfacing their values as `any`. It is the only type-erased seam, and it deliberately exposes no value.
Every Node[T] satisfies it, so a slice of unrelated node types can be started in one loop and then collected individually through their typed handles.
type Scope ¶
type Scope struct {
// contains filtered or unexported fields
}
Scope is one orchestrator's level in the dependency-injection chain. A scope sees its own providers and its ancestors' — never its descendants' or a sibling subtree's privates. Visibility runs strictly downward, which is what "information flows downward" means concretely.
A scope also carries the context that owns every node built under it. This is the lifetime boundary: cancelling a scope cancels its nodes; a single consumer's cancellation never reaches them.
A Scope is safe for concurrent use: providers may be registered and resolved from different goroutines. The recommended discipline is still to do all wiring during graph construction, before anything is demanded, so a missing provider fails before any work starts (see MustResolve).
func NewScope ¶
func NewScope(ctx context.Context, parent *Scope, opts ...ScopeOption) *Scope
NewScope creates a root scope whose context owns every node built under it. Pass nil as parent for a self-rooted scope (nothing resolves above it); pass a parent to chain onto an existing scope, which is what Scope.Child does.
Cancelling ctx is how a graph is torn down: every node owned by this scope (and by its descendants, if they derive their contexts from ctx) observes the cancellation in its body.
func (*Scope) Child ¶
func (s *Scope) Child(ctx context.Context, opts ...ScopeOption) *Scope
Child opens a sub-scope. Pass a derived context to make the child cancellable independently of its parent (that is how an orchestrator opts into fail-fast over the nodes IT owns, without touching shared siblings). Pass s.Context() to give the child exactly its parent's lifetime.
The child inherits the parent's trace observer; pass WithObserver to replace it for this subtree, or WithObserver(nil) to silence it.
type ScopeOption ¶ added in v0.2.0
type ScopeOption func(*scopeConfig)
ScopeOption configures a scope as it is created. Options are accepted by NewScope and Scope.Child.
func WithObserver ¶ added in v0.2.0
func WithObserver(fn func(Event)) ScopeOption
WithObserver installs fn as the trace observer for this scope: every node built under it reports [Event]s as it is demanded, starts, and finishes. Child scopes inherit the observer, so installing one at the root observes the whole graph.
This is how a lazy graph becomes debuggable. Without it, the interesting facts — what ran, what was skipped, how long each node took, what overlapped, how many consumers shared a value — are invisible by construction. With it, a test can assert "demanding the summary did not run the classifier" directly instead of inferring it, and a timing flag becomes a per-node profile.
Three parts of the contract matter:
fn is called from whichever goroutine reaches the moment — the demanding one for EventDemanded, the node's own for EventStarted and EventFinished — and from many of them at once. It MUST be safe for concurrent use.
fn MUST NOT block. EventFinished is delivered before the node's waiters unblock, which is what lets a test read a completed node's timing without racing it; the cost of that guarantee is that a slow observer slows every consumer of that node. Hand work off to a buffered channel rather than doing it inline.
Passing nil detaches: WithObserver(nil) on a child scope silences that subtree without affecting its parent. Nodes under an unobserved scope carry no observer at all, so tracing is genuinely zero-cost when unused.
Example ¶
WithObserver reports every node as it is demanded, starts and finishes — which is how you check that an undemanded branch really did not run.
package main
import (
"context"
"fmt"
"sync"
"github.com/heru-opensource/flowgraph"
)
func main() {
ctx := context.Background()
var mu sync.Mutex
ran := map[string]bool{}
demands := map[string]int{}
observe := func(e flowgraph.Event) {
mu.Lock()
defer mu.Unlock()
switch e.Kind {
case flowgraph.EventStarted:
ran[e.Label] = true
case flowgraph.EventDemanded:
demands[e.Label]++
case flowgraph.EventFinished:
}
}
// One call at the root; child scopes inherit the observer.
root := flowgraph.NewScope(ctx, nil, flowgraph.WithObserver(observe))
shared := flowgraph.Const(root, 21, flowgraph.Label("shared"))
wanted := flowgraph.Then(root, shared, func(ctx context.Context, v int) (int, error) {
return v * 2, nil
}, flowgraph.Label("wanted"))
flowgraph.Then(root, shared, func(ctx context.Context, v int) (int, error) {
return v * 100, nil
}, flowgraph.Label("skipped"))
v, _ := wanted.Get(ctx)
mu.Lock()
defer mu.Unlock()
fmt.Println("value:", v)
fmt.Println("wanted ran: ", ran["wanted"])
fmt.Println("skipped ran:", ran["skipped"])
fmt.Println("shared demanded:", demands["shared"], "times, ran once:", ran["shared"])
}
Output: value: 42 wanted ran: true skipped ran: false shared demanded: 1 times, ran once: true