cloudrig

README

CloudRig

A local Google Cloud environment for realistic, deterministic integration testing.

CloudRig gives you a connected GCP environment on your machine. Your real application code and GCP clients interact with realistic local services, while tests can deterministically control time, failures, and state.

                         CloudRig
                            │
       ┌────────────────────┼────────────────────┐
       │                    │                    │
       ▼                    ▼                    ▼
   GCP Services         Real Workloads       Test Controls
       │                    │                    │
       │                    │              ┌─────┼─────┐
       │                    │              ▼     ▼     ▼
 Storage / PubSub       Functions/GKE    Time  Fault  Fork
 Firestore / Tasks      Cloud Run
 Scheduler / Secrets
       │
       └────────── Connected Event-Driven ──────────┘

---

Quick Start

Install CloudRig from a checkout, start it, and point gcloud at your local GCP (Go 1.25+):

git clone https://github.com/monirz/cloudrig && cd cloudrig
go install ./cmd/cloudrig    # puts cloudrig on your PATH

cloudrig start &
. ./cloudrig-env.sh          # points gcloud at CloudRig, with no credentials

Now use it exactly like the real thing:

gcloud storage buckets create gs://demo
gcloud storage cp README.md gs://demo/
gcloud storage ls gs://demo
# gs://demo/README.md

Point the client libraries at the same port with the standard emulator variables:

export PUBSUB_EMULATOR_HOST=localhost:4599
export FIRESTORE_EMULATOR_HOST=localhost:4599

Prefer a local binary instead? make build produces ./cloudrig in the repo.

---

See It in Action

One workflow, end to end: an object landing in a bucket triggers a function that publishes to Pub/Sub, a worker enqueues a Cloud Task, and fast-forwarding the clock fires its retries. Run the whole thing with examples/pipeline/run.sh, or step through it:

# Start CloudRig with a virtual clock. It points functions at its own services
# automatically; SINK_URL is this app's own setting.
SINK_URL=http://localhost:8080/ cloudrig start --clock virtual &
. ./cloudrig-env.sh

# Provision a bucket, a Pub/Sub topic, and a task queue.
gcloud storage buckets create gs://intake
curl -X PUT localhost:4599/v1/projects/cloudrig-local/topics/orders -d '{}'
gcloud tasks queues create pipeline --location=us-central1 --max-attempts=3

# Deploy the pipeline: ingest (storage -> Pub/Sub), worker (Pub/Sub -> Cloud Task).
cloudrig fn deploy ingest --source ./examples/pipeline/ingest --trigger-bucket intake
cloudrig fn deploy worker --source ./examples/pipeline/worker --trigger-topic orders

# Drop an object in. It flows through the whole chain and enqueues a task.
echo '{"id":42}' > order.json
gcloud storage cp order.json gs://intake/

# Fast-forward time. The task fires and retries, deterministically.
cloudrig clock advance 1h
#   sink: task attempt      <- the retries, with no real waiting
#   sink: task attempt

Every stage is real: the functions run as processes, Pub/Sub and Cloud Tasks are live, and the clock is injected so the retries need no real waiting.

Why CloudRig?

• Connected and event-driven. A storage write triggers a function; a scheduler job publishes to Pub/Sub which triggers another. Whole workflows run locally, in one process.
• Deterministic. Time is injected, so a test reproduces exactly, with no flaky time.Sleeps.
• Built for testing. Travel through time, inject failures, and fork state.
• Real clients. gcloud, Terraform, and the Google client libraries work unchanged, over gRPC and REST.

---

How To

• Run CloudRig
• Use gcloud
• Connect a GCP client
• Create a Pub/Sub workflow
• Deploy and trigger a Function
• Use Cloud Storage triggers
• Use Cloud Tasks
• Use Scheduler
• Run GKE workloads
• Use Terraform / OpenTofu
• Configure CloudRig
• Run CloudRig in CI

---

Supported Services

CloudRig emulates the following services:

Service	Status	Notes
Cloud Functions	✅	Real local execution (Go & Node), HTTP & event triggers
Cloud Storage	✅	Buckets, objects, uploads, versioning, signed URLs
Pub/Sub	✅	Publish, streaming pull, ack/nack, redelivery
Firestore	✅	Documents, queries, transactions, over gRPC
Cloud Tasks	✅	Deferred HTTP work, dispatched on the clock
Cloud Scheduler	✅	Cron jobs firing HTTP or Pub/Sub, on the clock
Secret Manager	✅	Secrets, versions, the latest alias
Cloud Logging	✅	Write and read structured entries with a filter
Service Usage	✅	gcloud services enable/disable/list
Cloud Run	✅	Runs a real container (needs Docker)
GKE	✅	Real local Kubernetes (needs k3d or kind)

Full compatibility and limitations: service guides · what's not supported.

---

Testing

• Time Travel: fast-forward the clock; scheduled work fires instantly.
• Fault Injection: inject errors, latency, and timeouts over REST and gRPC.
• Fork State: branch a seeded environment per test case.
• In-Process Testing: one isolated emulator inside each Go test.

---

Tutorials

• Build your first Function
• Build an event-driven application
• Build a Storage → Function → Pub/Sub workflow
• Build a complete multi-service application
• Terraform + CloudRig
• GKE + CloudRig

---

Guides

• Time Travel
• Fault Injection
• Fork State
• In-Process Testing
• Configuration
• Authentication
• Networking

---

Architecture

How CloudRig serves every service on one port, with an injected clock: ARCHITECTURE.md.

---

CLI Reference

Every subcommand and flag: docs/cli.md.

---

Development

git clone https://github.com/monirz/cloudrig && cd cloudrig
make build              # -> ./cloudrig
make check              # build, vet, lint, gofmt, tests

See CONTRIBUTING.md for conventions.

---

Roadmap

Planned services, distribution, and testing features: ROADMAP.md.

---

Contributing

Contributions, bug reports and ideas are welcome. See CONTRIBUTING.md.

---

License

MIT. See LICENSE.

Documentation

Overview

Package cloudrig is a local emulator for Google Cloud APIs.

Start runs it as a server; MustStart runs it in-process inside a Go test:

func TestUpload(t *testing.T) {
	t.Parallel()
	emu := cloudrig.MustStart(t)
	// ... point a GCP client at emu.BaseURL()
}

MustStart is the reason the project exists: no container, no daemon, and one isolated instance per test.

Index

• Constants
• type Emulator
  • func MustStart(t testing.TB, opts ...Options) *Emulator
  • func Start(ctx context.Context, o Options) (*Emulator, error)
  • func (e *Emulator) BaseURL() string
  • func (e *Emulator) Clock() clock.Clock
  • func (e *Emulator) CloudRun() *cloudrun.Registry
  • func (e *Emulator) Endpoint() string
  • func (e *Emulator) FakeClock(t testing.TB) *clock.FakeClock
  • func (e *Emulator) Faults() *faults.Set
  • func (e *Emulator) Fork(t testing.TB) *Emulator
  • func (e *Emulator) FunctionURL(name string) string
  • func (e *Emulator) Functions() *functions.Registry
  • func (e *Emulator) Reset(ctx context.Context, project string) error
  • func (e *Emulator) Shutdown(ctx context.Context) error
  • func (e *Emulator) SyncEvents()
  • func (e *Emulator) SyncScheduler()
  • func (e *Emulator) SyncTasks()
• type Options

Constants

const DefaultAddr = ":4599"

DefaultAddr binds every interface so the binary is reachable from a container. MustStart ignores it and takes a random free loopback port.

Types

type Emulator

type Emulator struct {
	// contains filtered or unexported fields
}

Emulator is a running instance.

func MustStart

func MustStart(t testing.TB, opts ...Options) *Emulator

MustStart runs the emulator in-process for one test, on a random free port with a FakeClock and t.Cleanup registered. Every call is fully isolated.

func Start

func Start(ctx context.Context, o Options) (*Emulator, error)

Start runs the emulator on a real listener; the caller owns shutdown. ctx bounds startup only and does not stop the server.

func (*Emulator) BaseURL

func (e *Emulator) BaseURL() string

BaseURL is the endpoint as a URL: what STORAGE_EMULATOR_HOST and the SDK endpoint overrides want.

func (*Emulator) Clock

func (e *Emulator) Clock() clock.Clock

Clock returns the emulator's clock.

func (*Emulator) CloudRun

func (e *Emulator) CloudRun() *cloudrun.Registry

CloudRun is the registry of deployed Cloud Run services.

func (*Emulator) Endpoint

func (e *Emulator) Endpoint() string

Endpoint is the host:port a client should dial, e.g. "127.0.0.1:53412".

func (*Emulator) FakeClock

func (e *Emulator) FakeClock(t testing.TB) *clock.FakeClock

FakeClock returns the clock as a *clock.FakeClock so a test can Advance it, failing t if the emulator is running a real one.

func (*Emulator) Faults

func (e *Emulator) Faults() *faults.Set

Faults is the live fault-injection rule set.

A rule armed here fails matching requests before they reach any service, so a test can prove its own retry and error handling rather than hoping the path is exercised.

func (*Emulator) Fork

func (e *Emulator) Fork(t testing.TB) *Emulator

Fork returns a second emulator carrying a copy of this one's state.

Metadata is copied; object payloads are shared, because they are content-addressed and immutable — the same bytes under the same name. So a fork of a hundred gigabytes of objects costs the size of the metadata.

What is copied is state, not processes: deployed functions, armed faults and the clock do not travel. The fork gets its own port, its own event bus and its own fault set, and starts with no functions deployed.

func (*Emulator) FunctionURL

func (e *Emulator) FunctionURL(name string) string

FunctionURL is where the named function is served, or "" if it is not deployed. It returns the short form, valid for the default project and location; the prefixed form is also served.

func (*Emulator) Functions

func (e *Emulator) Functions() *functions.Registry

Functions is the registry backing this emulator, so a test can deploy into a running instance rather than only at Start.

func (*Emulator) Reset

func (e *Emulator) Reset(ctx context.Context, project string) error

Reset clears emulator state. An empty project clears everything.

func (*Emulator) Shutdown

func (e *Emulator) Shutdown(ctx context.Context) error

Shutdown stops the emulator; MustStart registers it with t.Cleanup.

func (*Emulator) SyncEvents

func (e *Emulator) SyncEvents()

SyncEvents waits for every event published so far to reach its subscribers.

Delivery is asynchronous so a write never waits on a trigger. A test that uploads an object and then asserts a function ran needs this, or it would have to poll.

func (*Emulator) SyncScheduler

func (e *Emulator) SyncScheduler()

SyncScheduler waits for every fired Cloud Scheduler HTTP job to finish, the same way SyncTasks does for tasks.

func (*Emulator) SyncTasks

func (e *Emulator) SyncTasks()

SyncTasks waits for every immediately-dispatched Cloud Tasks task to finish, including arming any retry. A test advances the clock for scheduled tasks; this covers the due-now ones that run off a goroutine.

type Options

type Options struct {
	// Addr is the listen address. Empty means DefaultAddr for Start, a random
	// free loopback port for MustStart.
	Addr string

	// Clock is the only source of time. Empty means real for Start, fake for
	// MustStart: a test that cannot control time will eventually sleep.
	Clock clock.Clock

	// Version is reported by /_emu/health. Empty means "dev".
	Version string

	// Runner is "auto", "subprocess" or "none". "auto" resolves to
	// "subprocess" when Functions is non-empty, else "none".
	Runner string

	// Functions are built and launched at Start, and stopped at Shutdown.
	Functions []functions.Function

	// FunctionLog receives function output. Nil discards it; use fn logs.
	FunctionLog io.Writer

	// EventLog receives the emulator's own messages, such as a watched
	// function redeploying. Nil discards them.
	EventLog io.Writer

	// DataDir persists Cloud Storage across restarts: metadata snapshots and
	// object content live under it. Empty keeps everything in memory and a
	// temp directory, which is what a test wants — state surviving a test is a
	// bug, not a feature.
	DataDir string
}

Options configures an Emulator. The zero value is valid.