faultline

README

Find the bugs in your distributed system before production does.

_{The etcd/raft harness with a bug switched on: go test fails a seed, the timeline page shows the crash, and the replay command fails the same way. Without the bug switch, 40,000 seeds passed.}

faultline is deterministic simulation testing (DST) for Go. It lets an AI coding agent test a distributed Go system inside go test. It finds the bugs that make distributed tests flaky. They appear only when messages are delayed or dropped, the network partitions, nodes crash or an fsync fails. The system runs in virtual time on a simulated network and disk, where faultline injects those faults. Every random choice comes from one seed, so a failing seed is a reproducible bug report: it replays the identical run. In v0.1.2 the code under test is written against faultline's node API, as Go callbacks with no goroutines, or is etcd/raft through the included harness.

An agent that changes distributed code gets:

• retries, timeouts and crash recovery tested under faults, with message delays of 1 to 5 ms by default that can reorder messages on a link;
• a check of the fix: the seed that failed passes after the change, and fails again if the change is reverted;
• a regression test: the failing seed, pinned in a test of its own;
• invariants as code: Go functions that faultline runs after every event.

From a failing seed to a pinned regression test, every step except the fix itself is a command to run or a file to read. People follow the same steps by hand. To start, give your agent Use faultline with an AI coding agent. It walks through the loop and ends with a block for the agent's AGENTS.md or CLAUDE.md. To learn faultline by hand, build the following example step by step in Get started.

⭐ If faultline is useful to you, a star on GitHub helps other Go developers find it.

Claude Code plugin

The faultline plugin for Claude Code turns the agent guide into a skill, an agent and two commands. Add its marketplace and install it from a Claude Code session:

/plugin marketplace add hmdsefi/faultline
/plugin install faultline@faultline

The skill loads when a distributed Go test is flaky or a faultline test fails, the agent summarizes a failing seed's artifact directory, and /faultline:replay and /faultline:pin replay a seed and pin it in a test. Use faultline with an AI coding agent describes each part and how to update the plugin.

Who it is for

faultline is for Go teams, and their agents, that build a distributed system: a consensus or replication library, a database, a queue, a controller. Code on the node API consists of boot functions, message handlers and timers that use the simulated network and disk. The etcd/raft harness ports one real system to that API. faultline is the wrong tool when:

• You need to test existing goroutine, net or os code unchanged. It needs a port to the node API first.
• You want to test the deployed system, or inject faults into production as chaos engineering tools do. faultline simulates the machines, the network and the disks inside one test process.
• You want performance numbers. A callback takes zero virtual time.

Limitations and FAQ lists what v0.1.2 does not do, and how faultline relates to FoundationDB's simulation testing, TigerBeetle's VOPR, Antithesis, Jepsen and testing/synctest.

Example

The following test from the tutorial runs a key-value server n1 and a client c1 for 10 seconds of virtual time. It crashes the server about once a second and checks after every event that no acknowledged put is lost:

var kvOptions = faultline.Options{Seeds: 50, Duration: 10 * time.Second}

func TestKV(t *testing.T) { faultline.Run(t, kvOptions, kvWorld) }

func kvWorld(w *faultline.World) {
	s := &kv.Server{W: w}
	n1 := w.AddServer("n1", s.Boot)
	if err := w.Disk.Volume(n1).WriteFileDurable("/wal", nil); err != nil {
		w.T().Fatal(err)
	}
	c := &kv.Client{W: w, Server: n1.ID(), Puts: 200}
	w.AddClient("c1", c.Boot)

	w.Plan(&fault.Random{MaxDown: 1, Rules: []fault.Rule{{
		Kind: fault.KindCrash, Every: time.Second,
		MinFor: 10 * time.Millisecond, MaxFor: 500 * time.Millisecond,
	}}})
	w.Invariant("acked puts survive", func() error { return c.Lost(s, n1) })
	w.Final("every put acked", c.Done)
}

The tutorial builds kv.Server and kv.Client on the node API. The server's message handler has a bug on purpose. It appends each put to the log and acknowledges it at once, while a timer syncs the log only every 5 ms:

s.W.Net.Handle(n, func(from kernel.NodeID, msg any) {
	p := msg.(Put)
	if _, err := wal.Append([]byte(p.Key + "=" + p.Value + "\n")); err != nil {
		return // no ack: the client retries
	}
	s.Data[p.Key] = p.Value
	s.W.Net.Send(n, from, Ack{Key: p.Key})
})

go test finds the bug:

--- FAIL: TestKV (0.04s)
    kv_test.go:14: faultline: running 50 seeds from base 0x75c66bf05fc438d4 (test name)
    --- FAIL: TestKV/seed=0x287372ab06f1482e (0.04s)
        faultline: invariant "acked puts survive" violated at t=5.873034010s (event 1455)
          k169=v169 was acknowledged, but n1 holds ""
        replay:    FAULTLINE_SEED=0x287372ab06f1482e go test -v -run '^TestKV$' example.com/kv
        artifacts: /tmp/faultline-501/example.com_kv/TestKV/287372ab06f1482e/
    kv_test.go:14: faultline: stopping after failing seed 0x287372ab06f1482e; 49 of 50 seeds not run (set Options.KeepGoing to run all)
FAIL
FAIL	example.com/kv	0.896s
FAIL

The seeds come from the test name, so every machine runs the same ones. The replay command runs the failing seed again and gets the identical run: the same check fails at the same virtual time and event number.

The artifact directory holds the report, the trace, a text timeline, the fault schedule, the client history and timeline.html. Read backward from the failure, and the timeline shows the server sending the acknowledgment for k169 and crashing 0.3 ms later, before its next sync:

etcd/raft

harness/etcdraft runs clusters of etcd/raft v3.7.0 inside faultline, with checks for raft's safety properties after every event. One call, etcdraft.Test, runs a whole test. It tests the raft library with the settings you choose, such as PreVote and CheckQuorum. It does not test your application: the state machine on top of raft is the harness's own key-value store. The harness's code is also a complete port of a real library to the node API.

On an Apple M4 Max, one seed of a three-server cluster, 60 seconds of virtual time under the default faults, took 68 ms on average. faultline ran 40,000 seeds of the harness, and no seed failed. Five bug switches break the harness on purpose, and the checks caught each one on the first seed. Test etcd/raft with the harness shows how to write such a test, and gives the configurations and timings of those runs.

Install

faultline needs Go 1.26 or newer. The first command adds faultline to your module, and the second adds the etcd/raft harness. The third installs the faultline command, which re-renders an artifact directory; your tests do not need it.

go get github.com/hmdsefi/faultline
go get github.com/hmdsefi/faultline/harness/etcdraft
go install github.com/hmdsefi/faultline/cmd/faultline@latest

The root module depends only on the standard library and gograph, which it uses for the network topology, the partition and quorum queries, and the diagram of a failing run. The harness is a module of its own, so etcd/raft and protobuf stay out of your module unless you add it.

Status

v0.1.2, early: the API may change before v1.0; artifact formats are versioned. The roadmap lists what the later phases add.

Documentation and help

Start with the index, or go to the page you need:

• The documentation index: the tutorial, guides, reference and concept pages.
• The Go API on pkg.go.dev.
• CONTRIBUTING and SECURITY: how to send a change, and how to report a vulnerability privately.

License

faultline is licensed under the Mozilla Public License 2.0. You can use it in any project, including closed-source ones. If you change faultline's own files and distribute them, those files stay under the MPL 2.0. You must make their source available and keep the copyright and license notices.

Documentation

Overview

Package faultline runs deterministic simulation tests inside go test. Run turns one test into many runs of a simulated world, one per seed. Each run builds nodes on a simulated network and disk, injects faults, and checks the invariants the test registers. Every random choice comes from the seed, so a failing seed replays exactly, as often as it takes to find and fix the bug.

A test passes Run its options and a function that builds one world. In the following test, a client pings a server every 100 ms while a fault planner crashes the server at random times. The final check asks for pongs after the faults stop:

var pingOptions = faultline.Options{Seeds: 50}

func TestPing(t *testing.T) {
	faultline.Run(t, pingOptions, pingWorld)
}

// pingWorld builds one world. Run calls it once for every run.
func pingWorld(w *faultline.World) {
	srv := w.AddServer("n1", func(n *kernel.Node) {
		w.Net.Handle(n, func(from kernel.NodeID, _ any) { w.Net.Send(n, from, "pong") })
	})
	var lastPong kernel.Time
	w.AddClient("c1", func(n *kernel.Node) {
		w.Net.Handle(n, func(kernel.NodeID, any) { lastPong = w.Sim.Now() })
		var ping func()
		ping = func() {
			w.Net.Send(n, srv.ID(), "ping")
			n.After(100*time.Millisecond, "ping", ping)
		}
		n.Post("ping", ping)
	})
	w.Plan(&fault.Random{MaxDown: 1, Rules: []fault.Rule{
		{Kind: fault.KindCrash, Every: time.Second, MaxFor: 500 * time.Millisecond},
	}})
	w.Final("pongs after recovery", func() error {
		if lastPong < w.RecoveryStart() {
			return fmt.Errorf("last pong at %s, recovery started at %s", lastPong, w.RecoveryStart())
		}
		return nil
	})
}

Code inside the world runs as callbacks on one goroutine, in virtual time. It must take time from the kernel (Node.Now, Node.After) and randomness from World.Rand or Node.Rand, and it must not start goroutines. A node's BootFunc runs again at every restart, so a node rebuilds its state there, from its disk if it has one. Packages kernel, simnet, simdisk and fault describe the event loop, the network, the disks and the faults.

Each seed runs as a subtest named seed=0x followed by 16 hex digits. By default the seeds come from the test's name (NameBase, DeriveSeed), so they are the same on every machine. When a seed fails, Run prints the failure and a command that replays that seed. With a server that stops answering after its first restart, TestPing fails like this:

--- FAIL: TestPing/seed=0x0fa36865fb584320 (0.02s)
    faultline: final check "pongs after recovery" failed after the run at t=60.000000000s
      last pong at 1.305834063s, recovery started at 45.000000000s
    replay:    FAULTLINE_SEED=0x0fa36865fb584320 go test -v -run '^TestPing$' example.com/ping

The line after the replay command names the artifact directory, by default under faultline-<uid> in os.TempDir(). Its files include report.txt (the lines above), trace.jsonl (every event of the run), schedule.json (the faults) and timeline.html (a page that shows the run). Package artifact describes them all. The replay command runs the seed again and gets the same run, record for record. That holds as long as the faultline version, the Go minor version, the code and the options stay the same.

To keep a seed as a regression test, run it with RunSeeds in a test of its own. RunSeeds runs only the seeds it lists. The following test gives it the options and world function of TestPing:

func TestPingSeed0fa36865(t *testing.T) {
	faultline.RunSeeds(t, pingOptions, pingWorld, 0x0fa36865fb584320)
}

Run's documentation lists the environment variables that choose seeds, replay a schedule and turn on the determinism check. The guides and reference pages are at https://github.com/hmdsefi/faultline/tree/main/docs.

Index

• Constants
• func DeriveSeed(base uint64, i int) uint64
• func NameBase(testName string) uint64
• func Run(t *testing.T, opts Options, body func(w *World))
• func RunSeeds(t *testing.T, opts Options, body func(w *World), seeds ...uint64)
• type Mode
  • func (m Mode) String() string
• type Options
• type World
  • func (w *World) AddClient(name string, boot kernel.BootFunc, opts ...kernel.NodeOption) *kernel.Node
  • func (w *World) AddServer(name string, boot kernel.BootFunc, opts ...kernel.NodeOption) *kernel.Node
  • func (w *World) Clients() []*kernel.Node
  • func (w *World) End() kernel.Time
  • func (w *World) Final(name string, check func() error)
  • func (w *World) Invariant(name string, check func() error)
  • func (w *World) Logf(format string, args ...any)
  • func (w *World) Options() Options
  • func (w *World) Plan(p fault.Planner)
  • func (w *World) Rand(label string) *rand.Rand
  • func (w *World) RecoveryStart() kernel.Time
  • func (w *World) Role(name string, fn func() []kernel.NodeID)
  • func (w *World) RunFor(d time.Duration)
  • func (w *World) Seed() uint64
  • func (w *World) Servers() []*kernel.Node
  • func (w *World) T() testing.TB

Examples

• DeriveSeed
• Run
• RunSeeds

Constants

const (
	DefaultSeeds     = 20               // Options.Seeds == 0
	DefaultDuration  = 60 * time.Second // Options.Duration == 0
	DefaultMaxEvents = 10_000_000       // Options.MaxEvents == 0
	// ShortSeeds caps the seed count under `go test -short` unless FAULTLINE_SEEDS is set.
	ShortSeeds = 5
	// MaxSeeds is the largest value FAULTLINE_SEEDS and Options.Seeds accept.
	MaxSeeds = 1_000_000
)

Defaults that Run applies to zero-valued Options fields.

Functions

func DeriveSeed

func DeriveSeed(base uint64, i int) uint64

DeriveSeed returns seed i (0-based) of the seed list derived from base with SplitMix64. It panics if i < 0.

Example

The seed list of a test depends only on its name, so the subtests of TestKV are the same on every machine. This prints the names of its first three seed subtests.

package main

import (
	"fmt"

	"github.com/hmdsefi/faultline"
)

func main() {
	base := faultline.NameBase("TestKV")
	for i := range 3 {
		fmt.Printf("TestKV/seed=0x%016x\n", faultline.DeriveSeed(base, i))
	}
}

Output:
TestKV/seed=0x287372ab06f1482e
TestKV/seed=0xac1b1f508c3fd743
TestKV/seed=0x9cb5a1e8588ac16e

func NameBase

func NameBase(testName string) uint64

NameBase returns the default base seed for a test name: FNV-1a 64 of the bytes of testName.

func Run

func Run(t *testing.T, opts Options, body func(w *World))

Run runs body for each seed of the seed list, one seed after another, each seed in its own subtest named "seed=0x%016x". Call it from the test goroutine of t, at most once per t.

Run calls body for every run of a seed, so body must build all the state it uses from w. A failing seed runs a second time to write its artifacts, and with determinism checking a passing seed runs twice. The runs of a seed must do the same thing.

A failing seed fails its subtest. It prints the failure, a replay command and its artifact directory, which holds report.json, report.txt, trace.jsonl, timeline.html and the other files of package artifact. The first failing seed stops the loop unless Options.KeepGoing is set or FAULTLINE_SEED_LIST names the seeds.

These environment variables change a run without code changes:

• FAULTLINE_SEED runs one seed. FAULTLINE_SEEDS, FAULTLINE_BASE_SEED and FAULTLINE_EXPLORE choose the seed list.
• FAULTLINE_SCHEDULE replays a schedule.json from an artifact.
• FAULTLINE_ARTIFACTS sets the artifact root, faultline-<uid> under os.TempDir() by default (faultline on Windows), or turns artifacts off with "off" in any letter case.
• FAULTLINE_CHECK_DETERMINISM=1 runs every passing seed twice and compares the trace hashes.
• FAULTLINE_TRACE=full keeps full traces and writes artifacts for passing seeds too, but a pass artifact replaces only another pass artifact.
• FAULTLINE_SEED_LIST runs exactly the listed seeds, and FAULTLINE_RESULTS appends one JSON line per seed to a file.

An invalid value fails t before any seed runs.

Example

A client pings a server every 100 ms while a fault planner crashes the server at random times. The final check asks for pongs after the faults stop. TestPing is an ordinary test function; this example compiles it but does not run it, since Run needs the *testing.T of a real test.

package main

import (
	"fmt"
	"testing"
	"time"

	"github.com/hmdsefi/faultline"
	"github.com/hmdsefi/faultline/kernel"
	"github.com/hmdsefi/faultline/kernel/fault"
)

func main() {
	// pingWorld builds one world. Run calls it once for every run of every seed.
	pingWorld := func(w *faultline.World) {
		// The server answers every ping. Its boot function runs again after each restart.
		srv := w.AddServer("n1", func(n *kernel.Node) {
			w.Net.Handle(n, func(from kernel.NodeID, _ any) { w.Net.Send(n, from, "pong") })
		})
		// The client sends a ping every 100 ms and notes when the last pong arrived.
		var lastPong kernel.Time
		w.AddClient("c1", func(n *kernel.Node) {
			w.Net.Handle(n, func(kernel.NodeID, any) { lastPong = w.Sim.Now() })
			var ping func()
			ping = func() {
				w.Net.Send(n, srv.ID(), "ping")
				n.After(100*time.Millisecond, "ping", ping)
			}
			n.Post("ping", ping)
		})
		// About once a second, crash the server for up to 500 ms.
		w.Plan(&fault.Random{MaxDown: 1, Rules: []fault.Rule{
			{Kind: fault.KindCrash, Every: time.Second, MaxFor: 500 * time.Millisecond},
		}})
		// After the run, check that pongs came back once the faults stopped.
		w.Final("pongs after recovery", func() error {
			if lastPong < w.RecoveryStart() {
				return fmt.Errorf("last pong at %s, recovery started at %s", lastPong, w.RecoveryStart())
			}
			return nil
		})
	}

	TestPing := func(t *testing.T) {
		faultline.Run(t, faultline.Options{Seeds: 50}, pingWorld)
	}
	_ = TestPing
}

func RunSeeds

func RunSeeds(t *testing.T, opts Options, body func(w *World), seeds ...uint64)

RunSeeds is Run for a fixed list of seeds, such as a regression test for a seed that failed. It runs exactly the listed seeds, in the order given, each in its own subtest named "seed=0x%016x". It runs all of them, even after one fails. No seeds, or a seed listed twice, fails t before any seed runs. As with Run, call it from the test goroutine of t, and call Run or RunSeeds at most once per t.

RunSeeds ignores Options.Seeds and Options.BaseSeed, and the variables that choose seeds: FAULTLINE_SEED, FAULTLINE_SEEDS, FAULTLINE_BASE_SEED, FAULTLINE_EXPLORE and FAULTLINE_SEED_LIST. It logs the ones that are set, and an invalid value still fails t. Every other option and variable works as in Run.

The replay command of a failing seed selects the seed's subtest with -run instead of setting FAULTLINE_SEED:

go test -v -run '^TestKVSeed287372ab$/^seed=0x287372ab06f1482e$' example.com/kv

Like Run, RunSeeds seeds crypto/rand for every run, which Go does not allow in a parallel test. A test that calls RunSeeds can call t.Parallel only with Options.NoCryptoSeed set.

Example

Seed 0x0fa36865fb584320 of TestPing failed, and the bug is fixed. TestPingSeed0fa36865 keeps that seed as a regression test: it runs only that seed, with the options and the world function of TestPing, and fails again if the bug comes back. Like ExampleRun, this example compiles the test but does not run it.

package main

import (
	"testing"

	"github.com/hmdsefi/faultline"
)

func main() {
	pingOptions := faultline.Options{Seeds: 50}
	// pingWorld builds one world, as in the example of Run.
	pingWorld := func(w *faultline.World) {
		// ...
	}

	TestPing := func(t *testing.T) {
		faultline.Run(t, pingOptions, pingWorld)
	}
	TestPingSeed0fa36865 := func(t *testing.T) {
		faultline.RunSeeds(t, pingOptions, pingWorld, 0x0fa36865fb584320)
	}
	_, _ = TestPing, TestPingSeed0fa36865
}

Types

type Mode

type Mode uint8

Mode selects how code under test runs.

const (
	// ModeEvent runs code written as kernel callbacks (boot functions, timers, network handlers)
	// on the kernel's event loop. A run is exactly reproducible from its seed.
	ModeEvent Mode = iota
	// ModeGoroutine is reserved for running goroutine code inside a testing/synctest bubble. Run
	// rejects it with a setup error.
	ModeGoroutine
)

func (Mode) String

func (m Mode) String() string

String returns "event", "goroutine", or "Mode(N)" for other values.

type Options

type Options struct {
	// Seeds is the number of seeds to run. 0 means DefaultSeeds. Negative or more than
	// MaxSeeds is a setup error.
	// FAULTLINE_SEEDS overrides it; FAULTLINE_SEED and FAULTLINE_SEED_LIST replace the seed list.
	// RunSeeds ignores it.
	Seeds int

	// BaseSeed is the base from which the seed list is derived. 0 means FNV-1a 64 of t.Name().
	// FAULTLINE_BASE_SEED and FAULTLINE_EXPLORE override it. RunSeeds ignores it.
	BaseSeed uint64

	// Duration is the virtual length of each run. 0 means DefaultDuration. Negative is a setup error.
	Duration time.Duration

	// MaxEvents stops a run that executes more events (kind "limit" unless AllowLimit).
	// 0 means DefaultMaxEvents. math.MaxUint64 means no practical limit.
	MaxEvents uint64

	// Mode selects how code under test runs. ModeEvent, the zero value, is the only mode Run
	// accepts.
	Mode Mode

	// Net configures the simulated network. The zero value means simnet.DefaultConfig(), so a
	// network with no latency at all cannot be configured here. The closest is a config with one
	// non-zero field, for example Default.Latency = 1, which gives every message 1 ns of latency.
	Net simnet.Config

	// Disk configures the simulated volumes and is passed to simdisk.New unchanged. The zero
	// value behaves like simdisk.DefaultConfig().
	Disk simdisk.Config

	// Trace configures the first run of every seed. The zero value is TraceHash.
	// FAULTLINE_TRACE=full upgrades it to TraceFull and writes artifacts for passing seeds.
	Trace kernel.TraceConfig

	// CheckDeterminism runs every passing seed a second time and compares trace hashes.
	// FAULTLINE_CHECK_DETERMINISM can enable it but not disable it.
	CheckDeterminism bool

	// KeepGoing runs all seeds even after one fails. Without it, Run stops starting new
	// seeds after the first failing seed. Forced on by FAULTLINE_SEED_LIST and by RunSeeds.
	KeepGoing bool

	// NoCryptoSeed stops Run from seeding crypto/rand with cryptotest.SetGlobalRandom. crypto/rand
	// in code under test is then not deterministic. In exchange, the test may call t.Parallel,
	// which cryptotest.SetGlobalRandom forbids.
	NoCryptoSeed bool

	// AllowLimit makes hitting MaxEvents (or the kernel's MaxTime) a normal end of the run
	// instead of a failure. Final checks are skipped for such a run.
	AllowLimit bool

	// Procs is reserved for goroutine mode: GOMAXPROCS during its runs. Run rejects a non-zero
	// value with a setup error.
	Procs int

	// Drain is reserved for goroutine mode: the virtual time processes get after the run ends to
	// exit on their own. Run rejects a non-zero value with a setup error.
	Drain time.Duration

	// StallTimeout is reserved for goroutine mode: the wall-clock time without progress after
	// which a watchdog ends the test binary with a report. Run rejects a non-zero value with a
	// setup error.
	StallTimeout time.Duration

	// FailOnLeak is reserved for goroutine mode: it makes goroutines still alive after the drain
	// a failure. Run rejects true with a setup error.
	FailOnLeak bool

	// Swarm is reserved for swarm testing, where each seed also picks which fault kinds are on.
	// Run rejects true with a setup error.
	Swarm bool
}

Options configures Run and RunSeeds. The zero value is valid: 20 seeds of 60 s virtual time each, default network, default disks, hash-only tracing.

type World

type World struct {
	Sim     *kernel.Sim       // the event loop, virtual time and the trace
	Net     *simnet.Network   // the network, created from Options.Net
	Disk    *simdisk.Disks    // the volumes, created from Options.Disk
	Faults  *fault.Injector   // applies faults and records them for schedule.json
	History *history.Recorder // client operations, written to history.jsonl
	// contains filtered or unexported fields
}

World is one simulated world, built for one run of one seed. Run builds a new World for every run, and the function passed to Run adds nodes, plans faults and registers checks on it. Its methods must be called from the seed subtest's goroutine only.

func (*World) AddClient

func (w *World) AddClient(name string, boot kernel.BootFunc, opts ...kernel.NodeOption) *kernel.Node

AddClient adds a node tagged "client". It equals w.Sim.AddNode(name, boot, append(opts, kernel.WithTags("client"))...).

func (*World) AddServer

func (w *World) AddServer(name string, boot kernel.BootFunc, opts ...kernel.NodeOption) *kernel.Node

AddServer adds a node tagged "server". It equals w.Sim.AddNode(name, boot, append(opts, kernel.WithTags("server"))...).

func (*World) Clients

func (w *World) Clients() []*kernel.Node

Clients returns the nodes tagged "client", in NodeID order.

func (*World) End

func (w *World) End() kernel.Time

End returns the virtual time at which the run ends: the replayed schedule's End when non-zero, else kernel.Time(Options.Duration).

func (*World) Final

func (w *World) Final(name string, check func() error)

Final registers a check that runs once after the run ends normally. All final checks run, in registration order; the first failing one is the seed's failure (kind "final").

func (*World) Invariant

func (w *World) Invariant(name string, check func() error)

Invariant registers a check that runs after every executed event. It must be cheap and must not change the world. The first non-nil error fails the run (kind "invariant").

func (*World) Logf

func (w *World) Logf(format string, args ...any)

Logf records a global kernel.log record (w.Sim.Logf) and, in the seed's first run only, logs the message with t.Logf prefixed by the virtual time.

func (*World) Options

func (w *World) Options() Options

Options returns the effective options of this Run or RunSeeds call: defaults applied, environment overrides applied, Seeds set to the length of the seed list, BaseSeed set to the resolved base (0 when the list came from FAULTLINE_SEED, FAULTLINE_SEED_LIST or RunSeeds), Trace set to the config of the seed's first run. It returns the same value in every run.

func (*World) Plan

func (w *World) Plan(p fault.Planner)

Plan registers a fault planner. Planners start when the simulation first advances (the first World.RunFor, or after body returns). Under FAULTLINE_SCHEDULE no planner is started: the replayed schedule is the only source of faults.

func (*World) Rand

func (w *World) Rand(label string) *rand.Rand

Rand returns the PRNG stream "workload/<label>". label must not be empty.

func (*World) RecoveryStart

func (w *World) RecoveryStart() kernel.Time

RecoveryStart returns the start of the recovery window, where fault.Random planners end the faults they started and start no more. It is the replayed schedule's Recovery when non-zero, else the earliest Until of the registered *fault.Random planners, else End(). A planner's Until is End minus its Quiet, a quarter of the run by default. Final checks use RecoveryStart to ask whether the system recovered.

func (*World) Role

func (w *World) Role(name string, fn func() []kernel.NodeID)

Role registers a named node selector that planners can target (fault.Rule.Target).

func (*World) RunFor

func (w *World) RunFor(d time.Duration)

RunFor advances the simulation by d virtual time. It may only be called from body. If the run fails or hits a limit, RunFor does not return: body is stopped and Run continues with failure handling. Recovering that stop does not resume the run: every later RunFor, also a deferred one, stops body again.

func (*World) Seed

func (w *World) Seed() uint64

Seed returns the seed of this world.

func (*World) Servers

func (w *World) Servers() []*kernel.Node

Servers returns the nodes tagged "server", in NodeID order.

func (*World) T

func (w *World) T() testing.TB

T returns the seed subtest. Use it for t.Fatalf on setup errors; prefer Invariant, Final or Sim.Fail to report failures of the system under test.