lockstep

README

Write actors. Get exact replay.

lockstep is an actor framework for Go with deterministic simulation testing (DST) built in. You write typed actors with mailboxes, supervision, timers, and request and reply. The same actor code runs in your program on lockstep.New(), and in go test on a seeded simulation that partitions the network, drops and delays messages, and crashes and restarts nodes. Any failure in the simulation replays exactly from its seed, so a bug that depends on timing becomes one command you can rerun.

The actor model here is the one Erlang/OTP and Akka made familiar, and that Go libraries such as protoactor-go, Hollywood, Ergo, and GoAkt also use. lockstep's runtime owns every message delivery, timer, and I/O event, and that is what makes the simulation exact.

Features

lockstep includes:

• Typed actors and PIDs. Actor[M] receives one message type and PID[M] accepts only that type, so a message of the wrong type does not compile.
• Request and reply. Ask with a typed ReplyTo, timeouts, and Pipe, which delivers the reply as a message instead of blocking the actor.
• Bounded mailboxes with an explicit policy when full: fail the send, drop the newest or oldest message, or block the sender.
• Supervision trees in the Erlang style: one-for-one and all-for-one strategies, restart limits with backoff, death watch, an actor cap, and a Stopped hook on every stop and restart.
• Timers and effects on the runtime's clock: ctx.After, ctx.Every, and I/O as named effects with lockstep.Do.
• A pooled production engine. lockstep.New() runs actors on a fixed pool of worker goroutines, so turns of different actors run in parallel while each actor handles one message at a time.
• Overload protection in the production engine: new actors and user messages are refused while the live heap is near its memory limit, a turn that runs too long is reported and its actor isolated, and a growing goroutine count is logged.
• Logs and metrics. The runtime logs failures, restarts, shutdown, and overload events to a log/slog logger, and every actor has a logger of its own. Counters and gauges go to a backend that you implement, for example for Prometheus, OpenTelemetry, or expvar.
• Event-sourced persistent actors. persist.Spawn starts an actor whose state is rebuilt from a journal of events. A reply sent from a Then callback goes out only after its events are fsynced, and a restart recovers from the newest snapshot and the events after it. persist/filestore is the reference journal and snapshot store. The same actor code runs on the operating system's files in production and on a simulated disk in tests, where crashes, torn writes, and failed syncs exercise the real recovery code.
• Delivery helpers. lockstep/dedup holds the state that handling duplicate and stale messages needs: a Window of idempotency keys and tombstones, a Replies cache of first replies, and a Versions gate per entity. Each value is bounded by age and count, and a persistent actor keeps it in its snapshots.
• Deterministic simulation testing in lockstep/sim, built on faultline: multiple nodes, network partitions, message drops and delays, crashes and restarts, memory pressure, and exact replay from a seed.
• Deadlock and leak detection in simulation. A cycle of pending asks fails the seed with lockstep: deadlock: A → B → A. An actor that stops with an effect pending, a resource unclosed, or a file still open on the simulated disk fails it with a leak failure.
• A static analyzer, lockstep-vet, also available as a golangci-lint plugin. It reports time.Now, global randomness, goroutines, select, map ranges, and blocking calls in actor code, and it reports messages that share memory with their sender, actors that share state, ignored replies and errors, unhandled message types, and resources an actor never releases.
• A Raft example with three planted bugs that a seeded partition or crash exposes.

Install

lockstep needs Go 1.26 or later:

go version
go get github.com/hmdsefi/lockstep@v0.2.0

Quick start: a first actor

An actor is a type with a Receive method for one message type. The following program is a counter: Add messages change its total, and a Get message asks for it.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/hmdsefi/lockstep"
)

// Msg is every message the counter accepts.
type Msg interface{ isMsg() }

// Add adds N to the total.
type Add struct{ N int }

// Get asks for the total.
type Get struct{ Reply lockstep.ReplyTo[int] }

func (Add) isMsg() {}
func (Get) isMsg() {}

// Counter is an actor that keeps a running total.
type Counter struct{ total int }

func (c *Counter) Receive(ctx *lockstep.Context[Msg], msg Msg) {
	switch m := msg.(type) {
	case Add:
		c.total += m.N
	case Get:
		m.Reply.Send(ctx, c.total)
	}
}

func newCounter() lockstep.Actor[Msg] { return &Counter{} }

func getTotal(r lockstep.ReplyTo[int]) Msg { return Get{Reply: r} }

func main() {
	sys := lockstep.New()
	counter, err := lockstep.Spawn(sys, "counter", newCounter)
	if err != nil {
		log.Fatal(err)
	}
	for _, n := range []int{1, 2, 3} {
		if err := lockstep.Send[Msg](sys, counter, Add{N: n}); err != nil {
			log.Fatal(err)
		}
	}
	total, err := lockstep.Ask(sys, counter, getTotal).Await(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("total:", total)

	if err := sys.Shutdown(context.Background()); err != nil {
		log.Fatal(err)
	}
}

go run .

total: 6

Spawn returns a PID[Msg], so the compiler rejects a message of the wrong type. Send takes [Msg] explicitly because Add{N: n} alone has type Add. Ask puts a ReplyTo[int] into the Get message and returns a *Reply[int]. Await blocks, so call it outside actors; inside an actor, lockstep.Pipe delivers the reply as a message instead.

Configure the production engine

lockstep.New() runs actor turns on a fixed pool of worker goroutines. Turns of different actors run in parallel, the turns of one actor never overlap, and messages from one sender to one receiver arrive in send order. Timers and effect results run on the same workers. Four options configure the engine:

• WithWorkers(n) sets the number of workers. The default is runtime.GOMAXPROCS(0), read when New runs.
• WithThroughput(n) sets how many user messages an actor processes in one turn before it yields its worker to other actors. The default is 32.
• WithActorThroughput(n) is a Spawn option that replaces that limit for one actor, for example a small n for an actor whose messages are expensive.
• WithEffectPool(workers, queue) bounds effects started with lockstep.Do: at most workers run at once, and at most queue more wait for a free slot. The defaults are 64 and 1024. A Do call that finds both full resolves with ErrOverloaded.

The quick start's main can begin like this instead:

sys := lockstep.New(
	lockstep.WithWorkers(4),
	lockstep.WithThroughput(64),
	lockstep.WithEffectPool(16, 256),
)
counter, err := lockstep.Spawn(sys, "counter", newCounter, lockstep.WithActorThroughput(256))
if err != nil {
	log.Fatal(err)
}

The options apply only to lockstep.New(). Simulation checks the values and then ignores them: every turn there processes one message, in an order the seed decides.

Protect the production engine from overload

lockstep.New() stops accepting new actors and user messages while the live heap is near its limit, reports a turn that runs too long, and warns when the process has too many goroutines. Three options tune this, and each has a default:

• WithAdmission(high, low) sets the memory thresholds in bytes.
• WithWatchdog(d) sets how long a turn may run before it counts as stuck.
• WithGoroutineWarning(n) sets the goroutine count that triggers a warning.

sys := lockstep.New(
	lockstep.WithAdmission(768<<20, 640<<20),
	lockstep.WithWatchdog(2*time.Second),
	lockstep.WithGoroutineWarning(50_000),
)

Memory admission

New samples the live heap before it returns and then every 100 ms. Without WithAdmission, the thresholds follow the Go memory limit (debug.SetMemoryLimit or GOMEMLIMIT): admission stops when the heap reaches 90% of the limit and resumes when it falls to 80%. With no memory limit set, memory never refuses work. WithAdmission(high, low) replaces the limit with absolute byte thresholds and needs 0 < low < high. WithAdmission(0, 0) turns memory admission off, even when a limit is set.

While admission is stopped:

• Spawn returns an error that matches ErrOverloaded. Restarts of existing actors are not refused.
• Each Send and Ask to a user actor is treated as if the target's mailbox were full, so the mailbox policy decides. Under FailSend, the error matches both ErrOverloaded and ErrMailboxFull. System messages, replies, timer messages, and effect results are not refused.
• Actors that called ctx.SubscribeOverload receive an Overloaded signal when admission stops and another when it resumes.

The system logs the change: admission refusing at Warn when it stops and admission resumed at Info when it resumes.

Stuck turns

A turn that never returns holds its worker, and the actor behind it stops answering. One watchdog goroutine reports every turn that has run for at least the timeout. The default, DefaultWatchdogTimeout, is 5 seconds, and WithWatchdog(0) turns the watchdog off.

The report is one log line at Error with the actor's path and the stack of the worker goroutine that runs the turn. From then until the turn returns, the actor counts as stuck. User messages to it get its mailbox policy as if the mailbox were full, and under FailSend the error matches ErrMailboxFull. System messages, such as Stop, still queue, and a Stop that queued while the turn was stuck stops the actor after the turn returns, without a restart. Otherwise, when the turn returns, the incarnation fails with an error that matches ErrTimeout, and its supervisor decides whether to restart, stop, or escalate.

lockstep cannot preempt a turn. A turn that never returns keeps its worker, and Shutdown with a deadline returns the context's error when the deadline passes.

Goroutine count

Once a second, the system counts the goroutines of the whole process, not only lockstep's. When the count reaches the bound from WithGoroutineWarning(n), the system logs one warning at Warn with the count and a summary of the ten functions that hold the most goroutines. The default, DefaultGoroutineWarning, is 10,000, and WithGoroutineWarning(0) turns the warning off. After a warning, the next one waits until the count falls below 80% of the bound and reaches it again.

Logs and invalid values

These events go to the system logger, each with a node attribute. Without WithLogger, the system logger is the one slog.Default() returned when New ran, so set the default logger before you call New. Logs and metrics describes WithLogger. A negative watchdog timeout or goroutine bound, or a non-zero WithAdmission pair other than 0 < low < high, makes New panic with an error that wraps ErrInvalidOption.

Simulation checks these values and then ignores them. It stays deterministic, and overload there comes from sim.Options.Budget and s.Pressure, described in Architecture. The actor cap, WithMaxActors, applies in both engines.

Logs and metrics

lockstep.New() writes the runtime's events to a log/slog logger, gives every actor a logger of its own, and reports counters and gauges to a metrics backend that you provide. Neither adds a dependency to your module.

WithLogger sets the system logger. Without it, the system logger is slog.Default() as it is when New runs, so a later slog.SetDefault does not change a running system. To discard the logs, pass slog.New(slog.DiscardHandler).

Inside an actor, ctx.Logger() returns the system logger with the actor's node, path, and uid attributes. ctx.Logf logs a formatted line at Info through it, and the record's source is the line that called Logf. The quick start's counter can log each addition, with log/slog and os added to its imports:

// In main:
logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
sys := lockstep.New(lockstep.WithNodeName("counter-1"), lockstep.WithLogger(logger))

// In Counter.Receive, after c.total += m.N:
ctx.Logger().Info("added", "n", m.N, "total", c.total)

Each record carries node, path, and uid before n and total.

The runtime logs its own events through the same logger: failures and restarts, shutdown, and the overload events above. The Debug events are a full mailbox, a shed message, a dropped message, and a dead letter.

WithMetrics takes a value that implements lockstep.Metrics. lockstep ships no adapter, so you write one for the backend you use, such as Prometheus, OpenTelemetry, or expvar. Its Counter method registers a counter series and returns a lockstep.Counter, and its GaugeFunc method registers a gauge function. Both take the series' labels as lockstep.MetricLabel values. Metrics in the architecture page shows the whole interface.

New registers every series with the backend before it returns, and each series carries a node label first. lockstep adds to the counters as events happen, and the backend calls the gauge functions when it collects. Without WithMetrics, nothing is registered or counted.

In simulation, ctx.Logger() and ctx.Logf write to the node's kernel log. faultline gives each simulated node a kernel log, and its records go into trace.jsonl and the trace hash. The same seed therefore gives the same lines. Metrics are off there, and WithLogger and WithMetrics in sim.Options.System have no effect.

Architecture lists every event with its level and attributes, every counter and gauge, and an example adapter.

Deterministic simulation testing: the same actor under faults

sim.Simulate runs a test body once per seed. Each run is a simulated world with virtual time and a simulated network. The following test runs the counter on server node n1, sends 20 Add messages from client node c1, and lets faultline crash and restart n1 at random times. A final check runs when the run ends:

package main

import (
	"fmt"
	"testing"
	"time"

	"github.com/hmdsefi/faultline"
	"github.com/hmdsefi/faultline/kernel/fault"
	"github.com/hmdsefi/lockstep"
	"github.com/hmdsefi/lockstep/sim"
)

func TestCounter(t *testing.T) {
	opts := sim.Options{Options: faultline.Options{Seeds: 50, Duration: 10 * time.Second}}
	sim.Simulate(t, opts, func(s *sim.Sim) {
		w := s.World()
		var counter lockstep.PID[Msg]
		s.Node("n1", func(sys *lockstep.System) {
			counter, _ = lockstep.Spawn(sys, "counter", newCounter)
		})
		s.Client("c1", func(*lockstep.System) {})
		w.Plan(&fault.Random{MaxDown: 1, Rules: []fault.Rule{
			{Kind: fault.KindCrash, Every: 8 * time.Second, MaxFor: time.Second},
		}})

		for range 20 {
			w.RunFor(250 * time.Millisecond)
			if err := sim.Send[Msg](s, "c1", counter, Add{N: 1}); err != nil {
				w.T().Fatal(err)
			}
		}
		w.RunFor(4 * time.Second) // faults stop at 7.5s, a quarter before the end
		total, err := sim.Ask(s, "c1", counter, getTotal)
		w.Final("every add counted", func() error {
			if err != nil {
				return err
			}
			if total != 20 {
				return fmt.Errorf("total is %d after 20 adds", total)
			}
			return nil
		})
	})
}

The test imports faultline directly, so add it to go.mod and run the test:

go mod tidy
go test .

--- FAIL: TestCounter (0.01s)
    counter_test.go:16: faultline: running 50 seeds from base 0x64b5629963419063 (test name)
    --- FAIL: TestCounter/seed=0x870af07a512d3866 (0.01s)
        faultline: final check "every add counted" failed after the run at t=10.000000000s
          total is 0 after 20 adds
        replay:    FAULTLINE_SEED=0x870af07a512d3866 go test -v -run '^TestCounter$' example.com/counter
        artifacts: $TMPDIR/faultline-501/example.com_counter/TestCounter/870af07a512d3866/
    counter_test.go:16: faultline: stopping after failing seed 0x870af07a512d3866; 48 of 50 seeds not run (set Options.KeepGoing to run all)
FAIL
FAIL	example.com/counter	0.330s
FAIL

The first seed passes and the second fails. The counter keeps its total in memory, so a crash loses it. The run's schedule.json artifact shows when: n1 crashed at t=4.87s, after 19 adds, and restarted at t=5.71s with a new counter. The 20th add reached a node that was down.

The seeds come from the test's name, so they are the same on every machine. The replay line is the command that runs the failing seed again:

FAULTLINE_SEED=0x870af07a512d3866 go test -v -run '^TestCounter$' example.com/counter

=== RUN   TestCounter
    counter_test.go:16: faultline: FAULTLINE_SEED=0x870af07a512d3866: running 1 seed
=== RUN   TestCounter/seed=0x870af07a512d3866
    faultline: final check "every add counted" failed after the run at t=10.000000000s
      total is 0 after 20 adds
    replay:    FAULTLINE_SEED=0x870af07a512d3866 go test -v -run '^TestCounter$' example.com/counter
    artifacts: $TMPDIR/faultline-501/example.com_counter/TestCounter/870af07a512d3866/
--- FAIL: TestCounter (0.01s)
    --- FAIL: TestCounter/seed=0x870af07a512d3866 (0.01s)
FAIL
FAIL	example.com/counter	0.250s
FAIL

The replay fails the same check at the same virtual time, and its report.json holds the same trace hash (a hash of every event in the run) as the first run: 0x56d752b8033ec3c3. faultline writes the artifacts under os.TempDir(), shown here as $TMPDIR; they include trace.jsonl, timeline.txt, and a self-contained timeline.html.

Besides the checks you register, sim fails a seed when actors deadlock on each other's asks (lockstep: deadlock: A → B → A), when an effect has no handler, and when an actor stops or restarts with an effect still pending, a resource it owns failing to close, or a file it opened on the simulated disk still open.

Persistent actors: state that survives a restart

An event-sourced actor stores the events that produced its state. persist.Spawn starts one under a persistence ID, the name of its stream of events. A command handler returns events. lockstep writes them to a journal, calls fsync, and only then applies them to the state and runs the callbacks you registered with ctx.Then, so a reply sent from a callback tells the caller that the events are on disk. When the actor starts or restarts, it recovers its state from the newest snapshot and the events after it.

The following program is a bank account. A Deposit stores a Deposited event and replies with the new balance:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/hmdsefi/lockstep"
	"github.com/hmdsefi/lockstep/persist"
	"github.com/hmdsefi/lockstep/persist/filestore"
)

// Deposit adds Amount to the account and replies with the new balance.
type Deposit struct {
	Amount int64
	Reply  lockstep.ReplyTo[int64]
}

// Deposited is the event that a Deposit stores.
type Deposited struct{ Amount int64 }

// Account is an event-sourced actor whose state is its balance.
type Account struct{}

func (Account) Initial() int64 { return 0 }

func (Account) Handle(ctx *persist.Context[Deposit, Deposited, int64], _ int64, d Deposit) []Deposited {
	ctx.Then(func(ctx *persist.Context[Deposit, Deposited, int64], balance int64) {
		d.Reply.Send(ctx, balance) // runs after the event is committed
	})
	return []Deposited{{Amount: d.Amount}}
}

func (Account) Apply(balance int64, e Deposited) int64 { return balance + e.Amount }

func newAccount() persist.EventSourced[Deposit, Deposited, int64] { return Account{} }

func deposit(amount int64) func(lockstep.ReplyTo[int64]) Deposit {
	return func(r lockstep.ReplyTo[int64]) Deposit { return Deposit{Amount: amount, Reply: r} }
}

func main() {
	store, err := filestore.Open("data")
	if err != nil {
		log.Fatal(err)
	}
	sys := lockstep.New()
	account, err := persist.Spawn(sys, "account", "acct-1", newAccount, persist.Config[Deposited, int64]{
		Journal: store,
		Events:  persist.JSON[Deposited](),
	})
	if err != nil {
		log.Fatal(err)
	}
	balance, err := lockstep.Ask(sys, account, deposit(5)).Await(context.Background())
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("balance:", balance)

	if err := sys.Shutdown(context.Background()); err != nil {
		log.Fatal(err)
	}
}

Run it twice:

go run .
go run .

balance: 5
balance: 10

The first run creates the directory data. The second run starts a new system on the same directory and recovers the balance of 5 from the journal before it handles the new deposit.

Initial returns the state before the first event, Handle turns a command into events, and Apply folds one event into the state. Apply must be a pure function of its arguments, because recovery calls it again for every stored event. An actor handles one command at a time, and while it recovers and while a batch of events is being committed, its commands wait in the mailbox. The actor's message type is the command type, so Send and Ask reach it like any actor. persist.JSON encodes events with encoding/json. Any persist.Codec works whose Encode is deterministic and returns a slice it does not reuse afterwards.

To start recovery from a snapshot instead of from the first event, give the Config a snapshot store and a codec for the state. A *filestore.Store is both the journal and the snapshot store:

persist.Config[Deposited, int64]{
	Journal:       store,
	Events:        persist.JSON[Deposited](),
	Snapshots:     store,
	State:         persist.JSON[int64](),
	SnapshotEvery: 100,
	PruneEvents:   true,
}

The actor saves a snapshot after every 100 committed events, beside its commands. With PruneEvents, it deletes the events that an older snapshot covers. The newest two snapshots and the events after the older one stay, so a damaged newest snapshot falls back to the older one.

A failed write or commit fails the actor, with an error that wraps the store's error, and its supervisor decides what happens next. No reply goes out for events that did not commit, and a restart recovers the state from what the store holds.

The same actor on a simulated disk

filestore.Open uses the operating system's files. In a simulation, s.Volume("n1") returns the simulated disk of node n1 as a filestore.FS, and filestore.New runs a store on it. The disk is faultline's: it keeps what was synced when the node crashes, and Options.Disk chooses what a crash does to writes that were not synced (lose them, keep some, or tear one) and what a failed sync does to pending writes, sets the capacity, and adds latency. A planner makes syncs fail. The following test crashes and restarts n1 while c1 deposits 1, 20 times, and checks that no acknowledged deposit is lost. It adds to the program above, and like the first test it imports faultline directly:

package main

import (
	"fmt"
	"testing"
	"time"

	"github.com/hmdsefi/faultline"
	"github.com/hmdsefi/faultline/kernel/fault"
	"github.com/hmdsefi/faultline/kernel/simdisk"
	"github.com/hmdsefi/lockstep"
	"github.com/hmdsefi/lockstep/persist"
	"github.com/hmdsefi/lockstep/persist/filestore"
	"github.com/hmdsefi/lockstep/sim"
)

func TestAccount(t *testing.T) {
	opts := sim.Options{Options: faultline.Options{Seeds: 50, Duration: 20 * time.Second}}
	opts.Disk.Latency = simdisk.DefaultLatency()
	sim.Simulate(t, opts, func(s *sim.Sim) {
		w := s.World()
		var account lockstep.PID[Deposit]
		s.Node("n1", func(sys *lockstep.System) {
			store := filestore.New(s.Volume("n1"))
			account, _ = persist.Spawn(sys, "account", "acct-1", newAccount, persist.Config[Deposited, int64]{
				Journal: store,
				Events:  persist.JSON[Deposited](),
			})
		})
		s.Client("c1", func(*lockstep.System) {})
		w.Plan(&fault.Random{MaxDown: 1, Rules: []fault.Rule{
			{Kind: fault.KindCrash, Every: 4 * time.Second, MaxFor: time.Second},
		}})

		var acked int64
		for range 20 {
			w.RunFor(250 * time.Millisecond)
			if _, err := sim.Ask(s, "c1", account, deposit(1), lockstep.WithTimeout(500*time.Millisecond)); err == nil {
				acked++
			}
		}
		w.RunFor(10 * time.Second) // faults stop at 15s, a quarter before the end
		balance, err := sim.Ask(s, "c1", account, deposit(0))
		w.Final("no acknowledged deposit is lost", func() error {
			if err != nil {
				return err
			}
			if balance < acked {
				return fmt.Errorf("balance %d after %d acknowledged deposits", balance, acked)
			}
			return nil
		})
	})
}

The recovery that runs after every crash is the code that runs in production, on a disk that lost or tore the writes that were not synced. A store that acknowledged a deposit before its fsync could lose it in a crash, and the check would fail on the seed that finds it.

A Volume belongs to its node. Using it from another node, or from outside a node event, fails the seed with lockstep/misuse, and a file that an actor leaves open when it stops or restarts fails it with lockstep/leak. A Store closes every file before a call returns, so only code that opens files itself can leak one. Architecture describes the journal format, the fsync order, and how recovery tells a torn write from corruption.

Static analysis: lockstep-vet and golangci-lint

A replay is exact only if actor code takes time, randomness, and I/O from the runtime, and actors share nothing but messages. lockstep-vet runs nine analyzers, which report ten check codes:

• LS100: nondeterminism and blocking in actor code, such as time.Now, global math/rand, crypto/rand, a go statement, select, a range over a map, environment variables, channel operations, locks, and blocking network, file, database, or process calls.
• LS110: Reply.Await in actor code, which blocks the actor; use lockstep.Pipe instead.
• LS200: a variable used after it was sent. Once lockstep.Send, lockstep.Ask, or ReplyTo.Send returns, the receiver owns the slice, map, or pointer in the message, so the sender must copy it before sending or stop using it. lockstep-vet reports a use that the function's control flow can reach after the send, and only a use of the part that was sent: sending s.items and then reading s.count is fine. Assigning the variable again ends the tracking, unless the message holds the variable's own storage, such as its address or a slice of an array variable, where the assignment overwrites what was sent and is reported. On the branch where lockstep.Send returned an error, nothing was delivered, so the sender may keep using the message, for example to retry. A message type marked //lockstep:immutable is safe to send and keep.
• LS201: a type marked //lockstep:immutable that isn't. A field may hold basic values, error, time.Time, lockstep.PID, lockstep.Address, lockstep.ReplyTo, other immutable types and pointers to them, and arrays, slices, maps, and structs of those. Any other pointer, a channel, a function, an interface, a type parameter, and unsafe.Pointer are reported. No code may write into a slice or map field of the type or assign through a pointer to it, except to build a new value: writes to a local variable that the function initializes from T{...}, &T{...}, or new(T) and assigns nothing else are construction.
• LS210: shared mutable state. Actor code that writes a package variable is reported, and so is actor code that places the actor itself, a field that holds references, the address of a field, or a *lockstep.Context in a message. A field whose type is marked //lockstep:immutable may go in a message.
• LS220: the reply of lockstep.Ask that is never awaited or piped, and the error of lockstep.Send or lockstep.Spawn that is ignored.
• LS230: a message type with no case in Receive. When the actor's message type is an interface with an unexported method, every type of its package that implements it needs a case. So does every type the actor sends to itself through ctx.After, ctx.Every, lockstep.Do, or lockstep.Pipe. A case is a type in a type switch or a type assertion on the message.
• LS240: an actor field that holds a resource and is neither passed to ctx.Own nor released in Stopped. A resource is a field whose type implements io.Closer, a *time.Timer or *time.Ticker, or a type marked //lockstep:resource. A marked type with no Close, Stop, or Release method is reported too. A field holding a resource the actor does not own, such as a shared *sql.DB, takes //lockstep:allow LS240 <reason> in its doc comment.
• LS001 and LS002: malformed and unknown //lockstep: directives.

LS100, LS110, LS210, LS230, and LS240 look at actor code, except that LS240 also reports a //lockstep:resource type with no release method wherever the type is declared. LS200, LS201, and LS220 look at every function, because messages are sent, and immutable types written, from any function.

The markers //lockstep:immutable and //lockstep:resource go in the doc comment of a type declaration. A marker on an alias is misplaced, and //lockstep:immutable cannot mark an interface type. They work across packages: a type marked in one package is immutable or a resource in every package that imports it.

Install it and run it through go vet:

go install github.com/hmdsefi/lockstep/analysis/cmd/lockstep-vet@v0.2.0
go vet -vettool=$(which lockstep-vet) ./...

On the counter it prints nothing. If Receive also recorded time.Now() on every Add and waited on an ask with Await, it would print:

main.go:37:50: LS110: Reply.Await blocks the actor; use lockstep.Pipe to receive the reply as a message (actor code: (*Counter).Receive)
main.go:34:17: LS100: time.Now uses the wall clock; use ctx.Now, ctx.After or ctx.Every (actor code: (*Counter).Receive)

To accept one finding, add //lockstep:allow LS100 <reason> on the reported line, on the line above it, or in the doc comment of the enclosing declaration. The reason is required, and one directive can name several checks, as in //lockstep:allow LS200,LS210 <reason>. LS001 and LS002 can't be suppressed.

The same checks run in golangci-lint as the module plugin lockstep. Build a custom binary with this .custom-gcl.yml:

version: v2.14.0
plugins:
  - module: github.com/hmdsefi/lockstep/analysis
    import: github.com/hmdsefi/lockstep/analysis/golangci
    version: v0.2.0

Enable the linter in .golangci.yml:

version: "2"
linters:
  enable:
    - lockstep
  settings:
    custom:
      lockstep:
        type: module

Then run golangci-lint custom and ./custom-gcl run ./.... golangci-lint's module plugin guide describes both files.

Example: Raft under partitions and crashes

examples/raft is a Raft cluster written as lockstep actors. It runs three or five servers and their clients through partitions, crashes, paused servers, and lossy links, checks Raft's safety properties after every event, and replays any failing seed exactly. Three planted bugs show what a failing seed looks like.

Example: retries and cancellations

lockstep delivers each message at most once, in send order between one sender and one receiver, and never retries. Message delivery explains what your actors must handle themselves: a retried request that arrives twice, and a Cancel that arrives before its Create. Package dedup holds the state for that: a Window of keys and tombstones, a Replies cache of first replies, and a Versions gate per entity. examples/orders is an order service that keeps its delivery state in dedup types, and whose simulation tests find both bugs from a seed and pin the fixes.

Why the name "lockstep"

Marching in lockstep means moving in exact unison, every step identical. That's the promise: every run with the same seed is identical.

The name also comes from deterministic lockstep, a game-networking technique used by real-time strategy games such as Age of Empires and StarCraft. Instead of sending game state over the network, every machine starts from the same state, applies the same inputs in the same order, and runs a deterministic simulation, so every machine computes the same game.

lockstep's simulation mode has the same properties:

Deterministic lockstep (games)	lockstep (this project)
Same inputs, same order → same game state	Same seed → same message order → same system state
A replay file is tiny: only the inputs	A failing test comes down to a single seed
A desync means nondeterminism leaked in (floats, random numbers)	A divergence means nondeterminism leaked into an actor (time.Now, math/rand, a raw goroutine)
The engine must own every source of randomness	The runtime must own every scheduling, time, and I/O decision

What it doesn't mean: in games, lockstep peers wait for each other every tick. lockstep is not a game-sync library, and outside simulation it does not run in lockstep: no actor waits for another, and turns follow the order in which messages arrive. The lockstep behavior applies to simulation mode, where a single seeded scheduler steps every actor in a fixed, replayable order.

lockstep and faultline

lockstep's simulation mode, the lockstep/sim package, runs on faultline, deterministic simulation testing for Go. lockstep imports faultline; it isn't part of the faultline repo. Only lockstep/sim imports it, so a program that uses lockstep without sim never links faultline.

github.com/hmdsefi/lockstep          actor runtime
├── lockstep.New()                   production engine for your program
├── lockstep/persist                 event-sourced actors
│   └── persist/filestore            their journal and snapshot store, in files
├── lockstep/dedup                   state for duplicate and stale messages
└── lockstep/sim ──imports──▶ github.com/hmdsefi/faultline (v0.1.2)

persist, persist/filestore and dedup do not import faultline either. sim gives the first two a simulated disk through (*Sim).Volume.

Benchmarks

The bench module runs the same workloads on lockstep, protoactor-go, Hollywood, and GoAkt: ping-pong, fan-out, idle actors, send latency, and ask latency. AskPipe runs on lockstep only, and BenchmarkCalibrate runs no library at all: it sorts a fixed slice to measure the speed of the machine. This command runs them all:

go -C bench test -run '^$' -bench . ./...

SimSeeds measures how many seeds per second sim.Simulate runs. It is a test, TestSimSeeds, not a benchmark, and it appends its results to the file that LOCKSTEP_BENCH_OUT names. Run it in bench/:

LOCKSTEP_BENCH_OUT=$PWD/sim.txt go test -run '^TestSimSeeds$' .

The other three libraries are required only by bench/go.mod, so the root module never depends on them.

Lower is better in every row.

Result	lockstep	protoactor-go	Hollywood	GoAkt
Ping-pong, 1 pair (ns per message)	783	2,670	384	423
Ping-pong, GOMAXPROCS pairs (ns per message)	318	967	200	254
Fan-out to 1,024 consumers (ns per item)	521	287	369	229
100,000 idle actors (heap bytes per actor)	2,018	718	27,899	1,237
Spawning 100,000 actors and sending each a message (ns per actor)	2,547	1,485	6,740	48,141
Send latency, p50 (ns)	751	361	330	361
Send latency, p99 (ns)	2,415	1,698	1,243	942
Ask latency, 1 caller, p50 (ns)	2,049	3,712	1,723	1,072
Ask latency, 1 caller, p99 (ns)	10,809	18,505	9,648	5,897
Ping-pong, 1 pair (allocations per message)	1	1	2	0

Each number comes from a median of the run in bench/results/baseline.txt: commit 439c466, measured on 2026-10-31 with the CPU AMD EPYC 7763 64-Core Processor. bench/RESULTS.md has more results of that run, library by library.

A budget is a ceiling on one lockstep result. The budgets in bench/budgets.txt come from the baseline in bench/results/baseline.txt, which a continuous integration (CI) run on main measured on ubuntu-latest. Time budgets are ratios to another benchmark in the same run, so a slower runner moves both sides.

CI's bench workflow runs on every push to main and every week, and fails when a lockstep result goes over a budget. It also runs on pull requests that change code the benchmarks build: bench/, the root package, internal/, persist/, sim/ or the root go.mod. Other pull requests get a passing bench check without running the suite.

The job runs the benchmarks without GoAkt first (-skip '/lib=goakt'), then GoAkt's alone (-bench '/lib=goakt'). A stopped GoAkt system stays in memory and would change what later benchmarks measure. A single run of the command above is therefore not how the baseline was produced.

Limitations

lockstep runs local actors in one process, plus the multi-node simulation in lockstep/sim. It cannot preempt a turn, so the watchdog reports a stuck turn and isolates its actor but does not free the worker. There is no remoting or clustering, so actors on different nodes talk only in simulation, and lockstep ships no tracing or metrics adapter. The reference store in persist/filestore keeps files on a local disk; a journal or snapshot store of another kind implements persist.Journal and persist.SnapshotStore. Use one filestore.Store per directory: two Stores on the same directory, in one process or in two, are not detected and can overwrite each other's committed events.

The API can change before v1.0. The roadmap tracks what comes next.

Built with

gograph finds the cycles in the wait-for graph that lockstep/sim builds from pending asks, which is how it reports a deadlock.

License

lockstep is licensed under the Mozilla Public License 2.0.

Next steps

• Why lockstep: the kind of bug lockstep is built for, and when to use something else.
• Architecture: the runtime, its two engines, and how simulation maps onto faultline.
• The actor model: where actors came from, how they work, and what lockstep adds.

Documentation

Overview

Package lockstep is an actor runtime for Go whose actor code runs unchanged on the worker pool of New and on a seeded simulation in tests (package lockstep/sim), where a failing seed replays exactly.

A system built by New runs actor turns on a fixed pool of worker goroutines (WithWorkers): turns of different actors can run in parallel, while the turns of one actor never overlap. Effects started with Do run on a separate bounded pool (WithEffectPool).

Index

• Constants
• Variables
• func NewReply[R any](from Sender, opts ...AskOption) (ReplyTo[R], *Reply[R])
• func Pipe[R, M any](ctx *Context[M], r *Reply[R], to func(R, error) M)
• func Send[M any](from Sender, to PID[M], msg M) error
• func SendContext[M any](ctx context.Context, from Sender, to PID[M], msg M) error
• func Stop(from Sender, addr Address) error
• func Unwatch[M any](ctx *Context[M], addr Address)
• func Watch[M any](ctx *Context[M], addr Address) error
• type Actor
• type ActorEstimate
• type Address
  • func (a Address) IsZero() bool
  • func (a Address) Node() string
  • func (a Address) Path() string
  • func (a Address) String() string
  • func (a Address) UID() uint64
• type AskOption
  • func WithTimeout(d time.Duration) AskOption
• type ChildFailed
• type Context
  • func (c *Context[M]) After(d time.Duration, msg M) TimerID
  • func (c *Context[M]) CancelEffect(id EffectID) bool
  • func (c *Context[M]) CancelTimer(id TimerID) bool
  • func (c *Context[M]) Every(d time.Duration, msg M) TimerID
  • func (c *Context[M]) Fail(err error)
  • func (c *Context[M]) Logf(format string, args ...any)
  • func (c *Context[M]) Logger() *slog.Logger
  • func (c *Context[M]) Now() time.Time
  • func (c *Context[M]) Own(r io.Closer)
  • func (c *Context[M]) Parent() Address
  • func (c *Context[M]) Rand() *rand.Rand
  • func (c *Context[M]) Self() PID[M]
  • func (c *Context[M]) Stop()
  • func (c *Context[M]) SubscribeOverload()
  • func (c *Context[M]) UnsubscribeOverload()
• type Counter
• type Directive
  • func (d Directive) String() string
• type Effect
• type EffectID
  • func Do[T, M any](ctx *Context[M], e Effect[T], done func(T, error) M, opts ...EffectOption) EffectID
• type EffectOption
  • func WithEffectTimeout(d time.Duration) EffectOption
• type MailboxPolicy
  • func (p MailboxPolicy) String() string
• type MemSizer
• type MetricLabel
• type Metrics
• type Option
  • func WithAdmission(high, low uint64) Option
  • func WithDefaultMailbox(capacity int, policy MailboxPolicy) Option
  • func WithEffectPool(workers, queue int) Option
  • func WithGoroutineWarning(bound int) Option
  • func WithGuardianStrategy(s Strategy) Option
  • func WithLogger(l *slog.Logger) Option
  • func WithMaxActors(n int) Option
  • func WithMemoryEstimates() Option
  • func WithMetrics(m Metrics) Option
  • func WithNodeName(name string) Option
  • func WithThroughput(n int) Option
  • func WithWatchdog(d time.Duration) Option
  • func WithWorkers(n int) Option
• type OverloadReason
  • func (r OverloadReason) String() string
• type Overloaded
• type PID
  • func Lookup[M any](sys *System, path string) (PID[M], error)
  • func PIDAt[M any](node, path string) PID[M]
  • func Spawn[M any](parent Spawner, name string, factory func() Actor[M], opts ...SpawnOption) (PID[M], error)
  • func (p PID[M]) Address() Address
  • func (p PID[M]) IsZero() bool
  • func (p PID[M]) String() string
• type PanicError
  • func (e *PanicError) Error() string
  • func (e *PanicError) Unwrap() error
• type Reply
  • func Ask[M, R any](from Sender, to PID[M], build func(ReplyTo[R]) M, opts ...AskOption) *Reply[R]
  • func (r *Reply[R]) Await(ctx context.Context) (R, error)
• type ReplyTo
  • func (rt ReplyTo[R]) Fail(from Sender, err error)
  • func (rt ReplyTo[R]) Send(from Sender, v R)
• type Sender
• type Signal
• type SignalReceiver
• type Sizer
• type SpawnOption
  • func WithActorThroughput(n int) SpawnOption
  • func WithMailbox(capacity int, policy MailboxPolicy) SpawnOption
  • func WithStrategy(s Strategy) SpawnOption
• type Spawner
• type Starter
• type StopKind
  • func (k StopKind) String() string
• type StopReason
  • func Failed(err error) StopReason
  • func Restarting(err error) StopReason
  • func (r StopReason) String() string
• type Stopper
• type Strategy
  • func AllForOne(decider func(err error) Directive) Strategy
  • func OneForOne(decider func(err error) Directive) Strategy
• type System
  • func New(opts ...Option) *System
  • func (s *System) Estimates(n int) []ActorEstimate
  • func (s *System) Incarnation() uint32
  • func (s *System) Node() string
  • func (s *System) Shutdown(ctx context.Context) error
• type Terminated
• type TimerID

Examples

• WithMetrics

Constants

const DefaultEffectTimeout = 30 * time.Second

DefaultEffectTimeout is the timeout of an effect started without WithEffectTimeout.

const DefaultGoroutineWarning = 10_000

DefaultGoroutineWarning is the goroutine count above which a production System logs a warning.

const DefaultMaxActors = 1_000_000

DefaultMaxActors is the actor cap of a System built without WithMaxActors.

const DefaultWatchdogTimeout = 5 * time.Second

DefaultWatchdogTimeout is the stuck-turn timeout of a production System without WithWatchdog.

Variables

var (
	ErrStopped       = errors.New("lockstep: stopped")
	ErrNameTaken     = errors.New("lockstep: name taken")
	ErrWrongType     = errors.New("lockstep: wrong message type")
	ErrNotFound      = errors.New("lockstep: not found")
	ErrInvalidOption = errors.New("lockstep: invalid option")
	ErrInvalidName   = errors.New("lockstep: invalid name")
	ErrNotLocal      = errors.New("lockstep: address not local")
)

Errors of package lockstep.

var (
	ErrTimeout      = errors.New("lockstep: timeout")
	ErrTerminated   = errors.New("lockstep: actor terminated")
	ErrAwaitInActor = errors.New("lockstep: blocking wait inside an actor")
)

Errors of sending and asking.

var (
	Normal         = StopReason{Kind: StopNormal}
	ParentStopping = StopReason{Kind: StopParentStopping}
	Shutdown       = StopReason{Kind: StopShutdown}
	// NoActor is the reason of a Terminated for an address that was not alive when it was watched
	// (Erlang's noproc). It never reaches Stopped.
	NoActor = StopReason{Kind: StopNoActor}
)

var ErrActorCap = errors.New("lockstep: actor cap reached")

ErrActorCap is the error of a spawn refused because the System has reached its actor cap (WithMaxActors). The refusal's text names the parent, the live actor count, the cap and its reserve, and errors.Is matches ErrActorCap.

var ErrMailboxFull = errors.New("lockstep: mailbox full")

ErrMailboxFull is the error of a message that a full mailbox refuses. A refusal returns fmt.Errorf("%w: %s", ErrMailboxFull, address), so errors.Is matches it and the text names the target.

var ErrNoEffectHandler = errors.New("lockstep: no effect handler")

ErrNoEffectHandler is the error that fails a simulation seed when an actor starts an effect and the simulation has no handler for the effect's name. The failure wraps it, so errors.Is(err, ErrNoEffectHandler) holds, and its text names the effect and the actor.

var ErrOverloaded = engine.ErrOverloaded

ErrOverloaded is returned when the runtime refuses work: a spawn or a message while admission sheds load, or an effect when the effect queue is full. It is the same value in a System built by New and in simulation, so errors.Is(err, ErrOverloaded) holds for both.

Functions

func NewReply

func NewReply[R any](from Sender, opts ...AskOption) (ReplyTo[R], *Reply[R])

NewReply creates an ask without a target: a pending reply on from's system with Ask's timeout and exactly-once resolution, and the ReplyTo that answers it. The caller builds and sends the request itself with Send. An error from that Send (for example ErrMailboxFull) does not resolve the reply; the caller resolves it, for example with rt.Fail(from, err). A request that a DropNewest or DropOldest mailbox drops is not reported, so the reply resolves at its timeout. An invalid option value returns the zero ReplyTo and a reply resolved with an error wrapping ErrInvalidOption. NewReply panics if from is nil, an option is nil, or from is a Context outside its turn.

func Pipe

func Pipe[R, M any](ctx *Context[M], r *Reply[R], to func(R, error) M)

Pipe delivers the resolution of r to the actor of ctx as one of its own messages: when r resolves, to(value, err) is called during a turn of the same incarnation and its result is passed to Receive in that turn. If the incarnation has ended by then, nothing is delivered. Pipe panics if ctx, r or to is nil, if ctx is outside its turn, or if r was created on another system.

func Send

func Send[M any](from Sender, to PID[M], msg M) error

Send delivers msg to the actor to. It returns nil when msg was enqueued, became a dead letter, or was accepted by the transport for another node. Errors: ErrNotFound for the zero PID, ErrStopped when the sender's system is stopped, the target mailbox's policy error (ErrMailboxFull), the admission error (ErrOverloaded), and a synchronous transport error, wrapped. Send panics if from is nil or is a Context outside its turn.

func SendContext

func SendContext[M any](ctx context.Context, from Sender, to PID[M], msg M) error

SendContext is Send, except that when from is a *System, the target's policy is BlockSender and the message cannot be enqueued yet, it waits until the message is enqueued or ctx ends, and then returns ctx.Err(). Called from an actor's turn it never waits: BlockSender acts as FailSend. Send with a *System sender is SendContext with context.Background().

func Stop

func Stop(from Sender, addr Address) error

Stop asks the actor at addr to stop with reason Normal and returns without waiting. A parent may stop its child this way; any Sender may stop any local actor. A path address stops the actor registered at its path. Errors: ErrNotFound if addr is not alive or names a guardian, ErrNotLocal for another node's address (there is no remote stop). Stop panics if from is nil.

func Unwatch

func Unwatch[M any](ctx *Context[M], addr Address)

Unwatch cancels Watch. After it returns, the incarnation receives no Terminated for addr.

func Watch

func Watch[M any](ctx *Context[M], addr Address) error

Watch asks for a Terminated signal when the actor at addr stops for good. If addr is not alive, the signal is delivered with reason NoActor. A path address (PIDAt) watches the actor registered at its path at the time of the call. Errors: ErrNotFound for the zero address and guardian addresses; ErrNotLocal for an address on another node; ErrStopped when the calling actor is stopping or restarting (inside a restart's Stopped too).

Types

type Actor

type Actor[M any] interface {
	// Receive handles one user message. It runs during a turn and must not block.
	Receive(ctx *Context[M], msg M)
}

Actor is implemented by actor values. M is the actor's message type.

type ActorEstimate

type ActorEstimate struct {
	Addr         Address
	MailboxBytes int64
	StateBytes   int64
}

ActorEstimate is the memory estimate of one live actor.

type Address

type Address struct {
	// contains filtered or unexported fields
}

Address is the untyped address of one actor: node, path and uid. It is comparable; two addresses are equal when all three parts are equal. The zero Address names no actor.

func (Address) IsZero

func (a Address) IsZero() bool

IsZero reports whether a is the zero Address.

func (Address) Node

func (a Address) Node() string

Node returns the node name.

func (Address) Path

func (a Address) Path() string

Path returns the absolute path, for example "/user/orders/o-17".

func (Address) String

func (a Address) String() string

String returns "lockstep://<node><path>#<uid in lowercase hex>", or "<zero>" for the zero Address.

func (Address) UID

func (a Address) UID() uint64

UID returns the uid.

type AskOption

type AskOption func(*askConfig)

AskOption configures one Ask.

func WithTimeout

func WithTimeout(d time.Duration) AskOption

WithTimeout sets the ask's timeout. The default is 5 s. It never panics: Ask and NewReply resolve the reply with an error wrapping ErrInvalidOption when d <= 0.

type ChildFailed

type ChildFailed struct {
	Child     Address
	Err       error
	Directive Directive // the directive applied; 0 when Cleanup is true
	Cleanup   bool      // the failure happened in Stopped; no directive was applied
}

ChildFailed reports a child's failure after the supervisor's strategy has handled it.

type Context

type Context[M any] struct {
	// contains filtered or unexported fields
}

Context is an incarnation's handle to the runtime. The runtime passes the same *Context to every turn of one incarnation and a new one to the next incarnation. Self and Parent may be called at any time; every other method panics when called outside a turn of its incarnation. Call its methods only from the goroutine that runs the handler. Every method panics on a nil *Context.

func (*Context[M]) After

func (c *Context[M]) After(d time.Duration, msg M) TimerID

After delivers msg to the actor itself once, d from now on the engine clock. A negative d is treated as 0. The timer is cancelled if the incarnation ends first. Called from Stopped it schedules nothing and returns 0.

func (*Context[M]) CancelEffect

func (c *Context[M]) CancelEffect(id EffectID) bool

CancelEffect cancels the pending effect id of this incarnation and reports whether it was pending; done is then never called for it. It returns false for 0, for an ID of another incarnation, and for an effect that already has an outcome.

func (*Context[M]) CancelTimer

func (c *Context[M]) CancelTimer(id TimerID) bool

CancelTimer stops timer id. It reports whether the timer was active. Once it returns, the actor receives no message from that timer, even one already in the mailbox.

func (*Context[M]) Every

func (c *Context[M]) Every(d time.Duration, msg M) TimerID

Every delivers msg to the actor itself every d, starting d from now, until the timer is cancelled or the incarnation ends. While a previous message of the timer is still in the mailbox, a firing delivers nothing. It panics if d <= 0. Called from Stopped it schedules nothing and returns 0.

func (*Context[M]) Fail

func (c *Context[M]) Fail(err error)

Fail marks the actor failed with err when the current handler returns. Only the first call per handler counts. It panics if err is nil.

func (*Context[M]) Logf

func (c *Context[M]) Logf(format string, args ...any)

Logf logs fmt.Sprintf(format, args...) at slog.LevelInfo through Logger, with the caller of Logf as the record's source. When the tracer is enabled it also records lockstep.log with the line. It formats the line once, and not at all when Logger is not enabled for slog.LevelInfo and the tracer is off.

func (*Context[M]) Logger

func (c *Context[M]) Logger() *slog.Logger

Logger returns the actor logger: the system logger with the attributes node, path and uid of this actor. It returns the same *slog.Logger for the actor's lifetime, across restarts. In simulation it writes to the node's kernel log instead: each record at slog.LevelInfo or above becomes a kernel.log record of the node, and records below Info are dropped.

func (*Context[M]) Now

func (c *Context[M]) Now() time.Time

Now returns the engine clock: wall time in production, the node's local clock in simulation, where faultline's clock drift and jumps apply.

func (*Context[M]) Own

func (c *Context[M]) Own(r io.Closer)

Own registers r to be closed after the incarnation's Stopped hook returns. Resources are closed in reverse registration order. It panics if r is nil.

func (*Context[M]) Parent

func (c *Context[M]) Parent() Address

Parent returns the parent's address: the user guardian's for actors spawned from a System.

func (*Context[M]) Rand

func (c *Context[M]) Rand() *rand.Rand

Rand returns the random source of the current incarnation. Every call in one incarnation returns the same *rand.Rand; a restart gets a new one. Use it only inside the actor's handlers.

func (*Context[M]) Self

func (c *Context[M]) Self() PID[M]

Self returns the actor's PID. It is the same for every incarnation of the actor.

func (*Context[M]) Stop

func (c *Context[M]) Stop()

Stop stops the actor with reason Normal when the current handler returns. No further user message is processed by this incarnation.

func (*Context[M]) SubscribeOverload

func (c *Context[M]) SubscribeOverload()

SubscribeOverload subscribes the current incarnation to Overloaded signals. Subscribing again has no effect. If the System is shedding, an Overloaded{Active: true} signal is queued at once.

func (*Context[M]) UnsubscribeOverload

func (c *Context[M]) UnsubscribeOverload()

UnsubscribeOverload cancels the subscription of the current incarnation.

type Counter

type Counter interface {
	// Add increases the counter by n. lockstep passes n >= 1.
	Add(n uint64)
}

Counter is one counter series.

type Directive

type Directive uint8

Directive is a supervisor's decision about a failed child.

const (
	DirectiveRestart  Directive = iota + 1 // end the incarnation and start a new one in place
	DirectiveStop                          // stop the child for good with Failed(err)
	DirectiveEscalate                      // fail the supervisor with the child's error
	DirectiveResume                        // keep the incarnation; the failing message is lost
)

func (Directive) String

func (d Directive) String() string

String returns "restart", "stop", "escalate", "resume", or "Directive(<n>)".

type Effect

type Effect[T any] interface {
	// Name returns the effect's stable name: 1 to 64 characters from [A-Za-z0-9._-], starting
	// with a letter or a digit. Simulation finds the effect's handler by this name, and traces
	// show it. Names starting with "lockstep." are reserved for lockstep's built-in effects.
	Name() string

	// Run performs the operation and returns its result. A System built by New calls it on a
	// goroutine of its own; simulation never calls it. Run should return soon after ctx is done,
	// which happens when the effect times out or is cancelled.
	Run(ctx context.Context) (T, error)
}

Effect is an operation that waits: a network call, file I/O, a database query, CPU work over a budget. An actor starts it with Do and gets its outcome back as a message.

type EffectID

type EffectID uint64

EffectID identifies one Do call within a System. IDs start at 1 and increase by one per Do call that starts an effect. 0 means that no effect was started.

func Do

func Do[T, M any](ctx *Context[M], e Effect[T], done func(T, error) M, opts ...EffectOption) EffectID

Do starts effect e for the actor of ctx and returns its ID. When the effect's outcome is delivered, a later turn of the actor calls done with the result, or with the zero T and ErrTimeout or ErrOverloaded, and passes done's return value to Receive. done runs on the actor's turn, never concurrently with Receive. If the incarnation ends or CancelEffect cancels the effect first, done is never called.

Do panics if e is nil, e.Name() is invalid, done is nil, or an option is invalid.

type EffectOption

type EffectOption func(*effectConfig)

EffectOption configures one Do call.

func WithEffectTimeout

func WithEffectTimeout(d time.Duration) EffectOption

WithEffectTimeout sets the effect's timeout, measured on the engine's clock from the Do call. Do panics if d is not positive.

type MailboxPolicy

type MailboxPolicy uint8

MailboxPolicy selects what happens when a message is sent to a full mailbox. A mailbox is full when it holds as many sent messages as its capacity. Timer messages, piped replies and effect results are never refused and never count toward capacity.

const (
	// FailSend refuses the message: Send returns an error wrapping ErrMailboxFull. The default.
	FailSend MailboxPolicy = iota
	// DropNewest drops the message being sent and records the drop. Send returns nil.
	DropNewest
	// DropOldest drops the oldest sent message in the mailbox, records the drop and enqueues the
	// new one. Send returns nil.
	DropOldest
	// BlockSender makes an outside sender (a *System) wait until the mailbox has room, the System
	// admits messages again (see WithAdmission) and the actor is not stuck (see WithWatchdog), or
	// until its context ends. For a sender inside an actor, and in simulation, which cannot make a
	// caller wait, it behaves as FailSend.
	BlockSender
)

func (MailboxPolicy) String

func (p MailboxPolicy) String() string

String returns "fail-send", "drop-newest", "drop-oldest", "block-sender", or "MailboxPolicy(<n>)" for other values. Records, logs and metrics use these spellings.

type MemSizer

type MemSizer interface{ MemSize() int }

MemSizer is implemented by actors that report the size of their state in bytes.

type MetricLabel

type MetricLabel struct {
	Name  string
	Value string
}

MetricLabel is one label of a series.

type Metrics

type Metrics interface {
	// Counter returns the counter for one series. lockstep keeps the result and calls its Add
	// method from any goroutine, including inside actor turns.
	Counter(name, help string, labels ...MetricLabel) Counter

	// GaugeFunc registers fn for one series. The backend calls fn whenever it collects, from any
	// goroutine. fn returns quickly and never blocks a Send. Once the system has shut down, fn
	// returns 0 and no longer refers to the system, except the goroutine count's, which keeps
	// reporting.
	GaugeFunc(name, help string, fn func() int64, labels ...MetricLabel)
}

Metrics is the backend lockstep reports to. Implementations must be safe for concurrent use. lockstep calls Counter and GaugeFunc once per series: from New, in a fixed order, and may call them later, from any goroutine, when a package enabled on the system after New registers series of its own (late registration). Duplicate registration (two systems with the same node name on one backend) is the implementation's to handle.

lockstep does not recover panics from Metrics, Counter or a gauge function.

type Option

type Option func(*systemConfig)

Option configures New. Options are applied in argument order; a later option overrides an earlier one for the same setting. Option values are validated by New, not by the option constructors: no option constructor panics.

func WithAdmission

func WithAdmission(high, low uint64) Option

WithAdmission sets absolute admission thresholds in bytes: admission stops when the live heap reaches high and resumes when it falls to low. WithAdmission(0, 0) turns admission off even when a Go memory limit is set. Any other pair needs 0 < low < high, or New panics with an error wrapping ErrInvalidOption. Simulation ignores it; its byte budgets are sim.Options.Budget.

func WithDefaultMailbox

func WithDefaultMailbox(capacity int, policy MailboxPolicy) Option

WithDefaultMailbox sets the capacity and policy of every actor spawned in the system without WithMailbox. Without it the default is 1024 and FailSend. It never panics: an invalid capacity or policy makes New panic with an error wrapping ErrInvalidOption.

func WithEffectPool

func WithEffectPool(workers, queue int) Option

WithEffectPool sets the effect pool: at most workers effects run at once, and at most queue more wait for a free slot. A Do call that finds both full resolves with ErrOverloaded. The defaults are 64 and 1024. New panics with an error wrapping ErrInvalidOption if workers < 1 or queue < 0.

func WithGoroutineWarning

func WithGoroutineWarning(bound int) Option

WithGoroutineWarning sets the goroutine count at which a production System logs a warning: one warning each time the process-wide count reaches the bound, with a summary of the functions that hold the most goroutines. The count is checked once per second, and the warning is armed again when it falls below the bound less a fifth of it. 0 turns the warning off. The count is also exported as the lockstep_goroutines gauge of WithMetrics. A bound < 0 makes New panic with an error wrapping ErrInvalidOption. Simulation ignores it.

func WithGuardianStrategy

func WithGuardianStrategy(s Strategy) Option

WithGuardianStrategy sets the user guardian's strategy for top-level actors. Without it, the strategy is OneForOne(nil). New panics with an error wrapping ErrInvalidOption if s is invalid.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger the system writes its log events to and that Context.Logger and Context.Logf build on. The default, and the value used when l is nil, is slog.Default() as it is when New runs. To discard logs, pass slog.New(slog.DiscardHandler).

func WithMaxActors

func WithMaxActors(n int) Option

WithMaxActors sets the maximum number of live actors of the System to n. n/100 of them (at least 1 when n >= 2) are reserved for restart rebuilds and system actors. An n < 1 makes New panic with an error wrapping ErrInvalidOption.

func WithMemoryEstimates

func WithMemoryEstimates() Option

WithMemoryEstimates turns on per-actor memory estimates (Sizer, MemSizer, System.Estimates).

func WithMetrics

func WithMetrics(m Metrics) Option

WithMetrics sets the metrics backend. New registers every counter and gauge function with m before it returns. nil, the default, disables metrics: nothing is registered or counted.

Example

ExampleWithMetrics reports a system's counters to a map. New registers every series with the backend before it returns, each with the node label first, and the system adds to a series each time its event happens.

package main

import (
	"context"
	"fmt"
	"strings"
	"sync"
	"sync/atomic"
	"time"

	"github.com/hmdsefi/lockstep"
)

// mapMetrics adapts Metrics to a set of counters kept in a map, keyed by the series in the form
// name{label="value",...}. A real backend registers each series with its metrics library instead.
type mapMetrics struct {
	mu       sync.Mutex
	counters map[string]*atomic.Uint64
}

// seriesKey returns the key of a series: its name and its labels, in order.
func seriesKey(name string, labels []lockstep.MetricLabel) string {
	parts := make([]string, len(labels))
	for i, l := range labels {
		parts[i] = fmt.Sprintf("%s=%q", l.Name, l.Value)
	}
	return name + "{" + strings.Join(parts, ",") + "}"
}

// mapCounter is one counter of a mapMetrics.
type mapCounter struct{ n *atomic.Uint64 }

func (c mapCounter) Add(n uint64) { c.n.Add(n) }

// Counter registers the counter of one series and returns it.
func (m *mapMetrics) Counter(name, _ string, labels ...lockstep.MetricLabel) lockstep.Counter {
	m.mu.Lock()
	defer m.mu.Unlock()
	n := new(atomic.Uint64)
	m.counters[seriesKey(name, labels)] = n
	return mapCounter{n}
}

// value returns the count of the series key, or 0 for a series that is not registered.
func (m *mapMetrics) value(key string) uint64 {
	m.mu.Lock()
	defer m.mu.Unlock()
	if n := m.counters[key]; n != nil {
		return n.Load()
	}
	return 0
}

// GaugeFunc ignores the gauge: this backend keeps counters only.
func (m *mapMetrics) GaugeFunc(string, string, func() int64, ...lockstep.MetricLabel) {}

type hello struct{}

type greeter struct{}

func (greeter) Receive(*lockstep.Context[hello], hello) {}

// ExampleWithMetrics reports a system's counters to a map. New registers every series with the
// backend before it returns, each with the node label first, and the system adds to a series each
// time its event happens.
func main() {
	m := &mapMetrics{counters: make(map[string]*atomic.Uint64)}
	sys := lockstep.New(lockstep.WithNodeName("example"), lockstep.WithMetrics(m))
	defer func() {
		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		_ = sys.Shutdown(ctx)
	}()

	if _, err := lockstep.Spawn(sys, "greeter", func() lockstep.Actor[hello] { return greeter{} }); err != nil {
		fmt.Println(err)
		return
	}

	const spawned = `lockstep_actors_spawned_total{node="example"}`
	fmt.Println(spawned, m.value(spawned))
}

Output:
lockstep_actors_spawned_total{node="example"} 1

func WithNodeName

func WithNodeName(name string) Option

WithNodeName sets the system's node name. The name must be 1-64 bytes from [A-Za-z0-9._-], starting with a letter or digit (the faultline node-name rule). WithNodeName never panics; New panics with an error wrapping ErrInvalidOption on an invalid name. The default is "local". It is ignored on a system built on an engine (simulation), which takes the engine's node name.

func WithThroughput

func WithThroughput(n int) Option

WithThroughput sets how many user messages one actor may process in a turn before it yields its worker to other actors. The default is 32. New panics with an error wrapping ErrInvalidOption if n < 1. It applies to the production engine only; simulation always uses 1.

func WithWatchdog

func WithWatchdog(d time.Duration) Option

WithWatchdog sets the stuck-turn timeout. A turn that runs at least d is logged and its actor is stuck until the turn returns: messages sent to it meanwhile meet its mailbox's policy as if the mailbox were full, so with FailSend, Send returns an error matching ErrMailboxFull. When a Started, Receive or Signal returns during the stuck turn, the actor fails with an error matching ErrTimeout, which its supervisor handles, unless that handler failed itself. A stuck Stopped, MemSize or factory only ends the stuck state. 0 turns the watchdog off. A d < 0 makes New panic with an error wrapping ErrInvalidOption. Simulation ignores it.

func WithWorkers

func WithWorkers(n int) Option

WithWorkers sets the number of worker goroutines that run actor turns. The default is runtime.GOMAXPROCS(0), read once when New runs. New panics with an error wrapping ErrInvalidOption if n < 1. The pool size never changes after New, even when GOMAXPROCS changes.

type OverloadReason

type OverloadReason uint8

OverloadReason is the source of an admission change. Its values equal those of engine.AdmissionSource.

const (
	OverloadMemory   OverloadReason = iota + 1 // production live heap
	OverloadBudget                             // simulation count budget (sim.Options.Budget)
	OverloadPressure                           // simulation memory-pressure fault
)

func (OverloadReason) String

func (r OverloadReason) String() string

String returns "memory", "budget", "pressure", or "OverloadReason(<n>)".

type Overloaded

type Overloaded struct {
	Active bool           // true: admission stopped; false: admission resumed
	Reason OverloadReason // the source whose change caused this transition
	Live   uint64         // measured bytes at the change (OverloadBudget: the estimate); 0 for OverloadPressure
	High   uint64         // thresholds in force (OverloadBudget: the byte budget and its resume mark); 0 for OverloadPressure
	Low    uint64
}

Overloaded is the signal sent to subscribed actors when their System starts or stops shedding. It implements Signal.

type PID

type PID[M any] struct {
	// contains filtered or unexported fields
}

PID is the typed address of an actor whose message type is M. It is comparable and contains only its Address. The zero PID names no actor.

func Lookup

func Lookup[M any](sys *System, path string) (PID[M], error)

Lookup returns the PID of the actor registered at path on sys, if it was spawned with exactly message type M. It finds an actor whose stop has begun and none whose stop has completed. It returns ErrInvalidName if path is not well-formed, ErrNotFound if no actor is registered there (the guardians /user and /system are not), and ErrWrongType if the actor's message type is not M. Lookup panics if sys is nil.

func PIDAt

func PIDAt[M any](node, path string) PID[M]

PIDAt returns a PID with uid 0 that addresses whichever actor is registered at path on node when the PID is used. A message to it when no actor of type M is registered at path goes to dead letters on that node. path has the form Lookup accepts. PIDAt panics if node is not a valid node name or path is malformed.

func Spawn

func Spawn[M any](parent Spawner, name string, factory func() Actor[M], opts ...SpawnOption) (PID[M], error)

Spawn creates a child of parent named name and returns its PID. name "" asks for a generated name "$<n>". factory is called at every start and restart, never by Spawn itself.

Errors: ErrInvalidName, ErrInvalidOption, ErrStopped, ErrNameTaken, ErrActorCap and ErrOverloaded, each wrapped so that errors.Is matches it. Spawn panics if parent, factory or an option is nil. A spawn refused for any of these reasons registers nothing and takes no generated name.

func (PID[M]) Address

func (p PID[M]) Address() Address

Address returns the PID's untyped address.

func (PID[M]) IsZero

func (p PID[M]) IsZero() bool

IsZero reports whether p is the zero PID.

func (PID[M]) String

func (p PID[M]) String() string

String returns p.Address().String().

type PanicError

type PanicError struct {
	Value any    // the value passed to panic
	Stack []byte // runtime/debug.Stack() at recovery; never recorded in traces
}

PanicError is the error of a failure caused by a panic. errors.Unwrap returns Value when it is an error.

func (*PanicError) Error

func (e *PanicError) Error() string

Error returns "lockstep: panic: " followed by a description of Value: Value.Error() for an error, Value.String() for a fmt.Stringer, the value itself for a string, integer or bool kind, and otherwise the type name ("nil" for a nil Value). The text never depends on a %v verb, so it is the same on every run.

func (*PanicError) Unwrap

func (e *PanicError) Unwrap() error

Unwrap returns Value if it is an error, otherwise nil.

type Reply

type Reply[R any] struct {
	// contains filtered or unexported fields
}

Reply is the typed future of one ask.

func Ask

func Ask[M, R any](from Sender, to PID[M], build func(ReplyTo[R]) M, opts ...AskOption) *Reply[R]

Ask sends the message build returns to the actor to and returns the future of its answer. build receives the ask's ReplyTo and is called once, synchronously, before Ask returns, unless the ask fails before sending. With a *System sender outside any actor's turn, a full BlockSender mailbox in a System built by New makes Ask wait until the request is enqueued or the ask resolves, at its timeout at the latest. An invalid option value resolves the reply with an error wrapping ErrInvalidOption. Ask panics if from or build is nil, an option is nil, or from is a Context outside its turn.

func (*Reply[R]) Await

func (r *Reply[R]) Await(ctx context.Context) (R, error)

Await waits for the ask to resolve and returns its value or error. If ctx ends first it returns the zero R and ctx.Err(), and the ask stays pending. Called during a turn it returns ErrAwaitInActor at once. In simulation Await never waits and never drives the run: on a pending reply it fails the seed and returns an error wrapping ErrAwaitInActor, from a turn or from the test body (test bodies use sim.Ask). Await may be called any number of times from any goroutine; every call that sees the resolution returns the same result.

type ReplyTo

type ReplyTo[R any] struct {
	// contains filtered or unexported fields
}

ReplyTo is the address an answer goes to. Ask creates it and passes it to build; the message carries it to whoever answers. It is a comparable value without pointers, so messages that hold it stay safe. The zero ReplyTo answers nothing.

func (ReplyTo[R]) Fail

func (rt ReplyTo[R]) Fail(from Sender, err error)

Fail answers the ask with err. It panics if err is nil, and like Send otherwise.

func (ReplyTo[R]) Send

func (rt ReplyTo[R]) Send(from Sender, v R)

Send answers the ask with v. The first reply attempt resolves the ask; later ones are dropped and traced. Send panics if from is nil or is a Context outside its turn.

type Sender

type Sender interface {
	// contains filtered or unexported methods
}

Sender is the origin of a send, an ask, a reply or a stop. It is implemented only by *System and *Context[X] for any X.

type Signal

type Signal interface {
	// contains filtered or unexported methods
}

Signal is a notification from lockstep itself, delivered through SignalReceiver[M].Signal and never as an M. It is implemented only by Terminated, ChildFailed and Overloaded.

type SignalReceiver

type SignalReceiver[M any] interface {
	Signal(ctx *Context[M], sig Signal)
}

SignalReceiver is implemented by actors that receive lockstep's notifications. An actor without it ignores signals; its supervisor strategy still handles its children's failures.

type Sizer

type Sizer interface{ Size() int }

Sizer is implemented by messages that report their size in bytes for memory estimates.

type SpawnOption

type SpawnOption func(*spawnConfig)

SpawnOption configures one Spawn. WithMailbox, WithStrategy and WithActorThroughput are SpawnOptions.

func WithActorThroughput

func WithActorThroughput(n int) SpawnOption

WithActorThroughput overrides WithThroughput for one actor: the actor processes at most n user messages per turn. Use a small n for an actor whose messages are expensive, a large one for a router that must drain quickly. Spawn returns an error wrapping ErrInvalidOption if n < 1. It has no effect in simulation, where every turn processes one message.

func WithMailbox

func WithMailbox(capacity int, policy MailboxPolicy) SpawnOption

WithMailbox sets the spawned actor's mailbox capacity and full-mailbox policy. It never panics: if capacity < 1 or policy is not one of the four constants, Spawn returns an error wrapping ErrInvalidOption.

func WithStrategy

func WithStrategy(s Strategy) SpawnOption

WithStrategy sets the strategy for the children of the spawned actor. Without it, the strategy is OneForOne(nil). Spawn returns an error wrapping ErrInvalidOption if s is invalid.

type Spawner

type Spawner interface {
	// contains filtered or unexported methods
}

Spawner is a parent for Spawn. It is implemented only by *System (the user guardian is the parent) and *Context[X] for any X (that actor is the parent).

type Starter

type Starter[M any] interface {
	Started(ctx *Context[M])
}

Starter is implemented by actors that need a hook when an incarnation starts. Started runs after the factory and before the first message of the incarnation.

type StopKind

type StopKind uint8

StopKind classifies a StopReason.

const (
	StopNormal         StopKind = iota + 1 // Stop(from, addr) or ctx.Stop
	StopFailed                             // the supervisor's directive was to stop a failed actor
	StopParentStopping                     // the parent is stopping or restarting
	StopShutdown                           // System.Shutdown
	StopRestarting                         // the incarnation ends for a restart; only Stopped sees it
	StopNoActor                            // Terminated only: the watched address was not alive
)

func (StopKind) String

func (k StopKind) String() string

String returns "normal", "failed", "parent-stopping", "shutdown", "restarting", "no-actor", or "StopKind(<n>)".

type StopReason

type StopReason struct {
	Kind StopKind
	Err  error
}

StopReason tells why an incarnation ended, or, in Terminated, why a watched actor is gone. Err is non-nil exactly for StopFailed and StopRestarting.

func Failed

func Failed(err error) StopReason

Failed returns a StopFailed reason carrying err: the reason of an actor stopped by its supervisor after failing with err. It panics if err is nil.

func Restarting

func Restarting(err error) StopReason

Restarting returns the reason Stopped receives when the incarnation ends for a restart caused by err. It panics if err is nil.

func (StopReason) String

func (r StopReason) String() string

String returns Kind.String(), followed by ": " and Err.Error() when Err is non-nil.

type Stopper

type Stopper[M any] interface {
	Stopped(ctx *Context[M], reason StopReason)
}

Stopper is implemented by actors that release resources. Stopped runs exactly once per incarnation: when the actor stops, before every restart, and after a failure, but not when a simulated node crashes. Children have already stopped when it runs.

type Strategy

type Strategy struct {

	// Decider maps a child's failure to a directive. nil means DirectiveRestart for every error.
	Decider func(err error) Directive
	// MaxRestarts is how many restarts are allowed within Window; one more becomes
	// DirectiveEscalate. 0 allows none.
	MaxRestarts int
	// Window is the sliding period over which restarts are counted, on the engine clock.
	Window time.Duration
	// MinBackoff is the delay before the first restart in the window; it doubles for each further
	// restart in the window, up to MaxBackoff. 0 restarts without delay.
	MinBackoff time.Duration
	MaxBackoff time.Duration
	// contains filtered or unexported fields
}

Strategy is a supervision strategy for the children of one actor. Build it with OneForOne or AllForOne and adjust the exported fields. A Strategy is valid when one of those built it, MaxRestarts and MinBackoff are not negative, Window is positive, and MaxBackoff is at least MinBackoff; a zero Strategy is invalid.

func AllForOne

func AllForOne(decider func(err error) Directive) Strategy

AllForOne returns a strategy that restarts or stops every child of the supervisor when one fails, with the same limits as OneForOne. Restarts are counted for the group.

func OneForOne

func OneForOne(decider func(err error) Directive) Strategy

OneForOne returns a strategy that applies a directive to the failed child only, with MaxRestarts 10, Window 1 minute, MinBackoff 10 ms and MaxBackoff 10 s.

type System

type System struct {
	// contains filtered or unexported fields
}

System is one runtime instance. All of its exported functions and methods are safe for concurrent use when *System is the Spawner or Sender.

func New

func New(opts ...Option) *System

New builds a system on a pool of worker goroutines that run actor turns, creates its two guardians and returns it. Turns of different actors can run in parallel on different workers; the turns of one actor never overlap. WithWorkers, WithThroughput and WithEffectPool configure the pools. The system also samples the live heap for memory admission (WithAdmission) and logs each turn that runs for at least the watchdog timeout at level Error, with the stack of the goroutine running it. The watchdog is on by default, with DefaultWatchdogTimeout; WithWatchdog sets the timeout or turns it off. New panics with an error wrapping ErrInvalidOption if an option is nil or invalid.

func (*System) Estimates

func (s *System) Estimates(n int) []ActorEstimate

Estimates returns up to n live actors with the largest estimates, largest first, ties in byte order of Addr.String(). It returns nil when n <= 0 or estimates are off.

func (*System) Incarnation

func (s *System) Incarnation() uint32

Incarnation returns the epoch the system started with: the faultline node incarnation in simulation, a random value otherwise. It is never 0 and does not change for the system's life, even when uid allocation moves to a new epoch.

func (*System) Node

func (s *System) Node() string

Node returns the system's node name: the node part of every address it allocates.

func (*System) Shutdown

func (s *System) Shutdown(ctx context.Context) error

Shutdown stops every actor, user actors first, then system actors, and waits until every Stopped hook has returned. It returns nil when the system is stopped, or ctx.Err() if ctx ends first; shutdown then continues and a later call waits again. Called during a turn (an actor may shut down its own system) or on an engine that cannot block, such as the simulation engine, it starts the shutdown and returns nil at once without waiting.

type Terminated

type Terminated struct {
	Addr   Address
	Reason StopReason
}

Terminated reports that a watched actor stopped for good, or that the watched address was not alive (Reason NoActor). Reason is the actor's final stop reason, or NoActor.

type TimerID

type TimerID uint64

TimerID identifies a timer of one actor. IDs are unique within the actor for its whole life, across restarts. 0 is never a valid timer.