e2e

package module
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 18, 2022 License: Apache-2.0 Imports: 21 Imported by: 8

README

e2e

golang docs

Go Module providing robust framework for running complex workload scenarios in isolation, using Go and Docker. For integration, e2e tests, benchmarks and more! 💪

What are the goals?

  • Ability to schedule isolated processes programmatically from a single process on a single machine.
  • Focus on cluster workloads, cloud native services and microservices.
  • Developer scenarios in mind - e.g. preserving scenario readability and integration with the Go test ecosystem.
  • Metric monitoring as the first citizen. Assert on Prometheus metric values during test scenarios or check overall performance characteristics.

Usage Models

There are three main use cases envisioned for this Go module:

  • e2e test use (see example). Use e2e in e2e tests to quickly run complex test scenarios involving many container services. This was the main reason we created this module. You can check usage of it in Cortex and Thanos projects.
  • Standalone use (see example). Use e2e to run setups in interactive mode where you spin up workloads as you want programmatically and poke with it on your own using your browser or other tools. No longer need to deploy full Kubernetes or external machines.
  • Benchmark use. Use e2e in local Go benchmarks when your code depends on external services with ease.
Getting Started

Let's go through an example leveraging the go test flow:

  1. Get the e2e Go module to your go.mod using go get github.com/efficientgo/e2e.

  2. Implement a test. Start by creating an environment. Currently e2e supports Docker environment only. Use a unique name for all of your tests. It's recommended to keep it stable so resources are consistently cleaned.

    	// Start isolated environment with given ref.
	e, err := e2e.New()
	testutil.Ok(t, err)
	// Make sure resources (e.g docker containers, network, dir) are cleaned.
	t.Cleanup(e.Close)

  3. Implement the workload by creating e2e.Runnable. Or you can use existing runnables in the e2edb package. For example implementing a function that schedules Jaeger with our desired configuration could look like this:

    	j := e.Runnable("tracing").
		WithPorts(
			map[string]int{
				"http.front":    16686,
				"jaeger.thrift": 14268,
			}).
		Init(e2e.StartOptions{Image: "jaegertracing/all-in-one:1.25"})

  4. Use e2emon.AsInstrumented if you want to be able to query your service for metrics, which is a great way to assess it's internal state in tests! For example see following Etcd definition:

    	return e2emon.AsInstrumented(env.Runnable(name).WithPorts(map[string]int{AccessPortName: 2379, "metrics": 9000}).Init(
		e2e.StartOptions{
			Image: o.image,
			Command: e2e.NewCommand(
				"/usr/local/bin/etcd",
				"--listen-client-urls=http://0.0.0.0:2379",
				"--advertise-client-urls=http://0.0.0.0:2379",
				"--listen-metrics-urls=http://0.0.0.0:9000",
				"--log-level=error",
			),
			Readiness: e2e.NewHTTPReadinessProbe("metrics", "/health", 200, 204),
		},
	), "metrics")

  5. Program your scenario as you want. You can start, wait for their readiness, stop, check their metrics and use their network endpoints from both unit test (Endpoint) as well as within each workload (InternalEndpoint). You can also access workload directory. There is a shared directory across all workloads. Check Dir and InternalDir runnable methods.

    	// Create structs for Prometheus containers scraping itself.
	p1 := e2edb.NewPrometheus(e, "prometheus-1")
	s1 := e2edb.NewThanosSidecar(e, "sidecar-1", p1)

	p2 := e2edb.NewPrometheus(e, "prometheus-2")
	s2 := e2edb.NewThanosSidecar(e, "sidecar-2", p2)

	// Create Thanos Query container. We can point the peer network addresses of both Prometheus instance
	// using InternalEndpoint methods, even before they started.
	t1 := e2edb.NewThanosQuerier(e, "query-1", []string{s1.InternalEndpoint("grpc"), s2.InternalEndpoint("grpc")})

	// Start them.
	testutil.Ok(t, e2e.StartAndWaitReady(p1, s1, p2, s2, t1))

	// To ensure query should have access we can check its Prometheus metric using WaitSumMetrics method. Since the metric we are looking for
	// only appears after init, we add option to wait for it.
	testutil.Ok(t, t1.WaitSumMetricsWithOptions(e2emon.Equals(2), []string{"thanos_store_nodes_grpc_connections"}, e2emon.WaitMissingMetrics()))

	// To ensure Prometheus scraped already something ensure number of scrapes.
	testutil.Ok(t, p1.WaitSumMetrics(e2emon.Greater(50), "prometheus_tsdb_head_samples_appended_total"))
	testutil.Ok(t, p2.WaitSumMetrics(e2emon.Greater(50), "prometheus_tsdb_head_samples_appended_total"))

	// We can now query Thanos Querier directly from here, using it's host address thanks to Endpoint method.
	a, err := api.NewClient(api.Config{Address: "http://" + t1.Endpoint("http")})
Interactive

It is often the case we want to pause e2e test in a desired moment, so we can manually play with the scenario in progress. This is as easy as using the e2einteractive package to pause the setup until you enter the printed address in your browser. Use the following code to print the address to hit and pause until it's getting hit.

err := e2einteractive.RunUntilEndpointHit()
Monitoring

Each instrumented workload (runnable wrapped with e2emon.AsInstrumented) have programmatic access to the latest metrics with WaitSumMetricsWithOptions methods family. Yet, especially for standalone mode it's often useful to query and visualise all metrics provided by your services/runnables using PromQL. In order to do so just start monitoring from e2emon package:

mon, err := e2emon.Start(e)

This will start Prometheus with automatic discovery for every new and old instrumented runnables. It also runs cadvisor that monitors docker itself if env.DockerEnvironment is started and shows generic performance metrics per container (e.g container_memory_rss). Run OpenUserInterfaceInBrowser() to open the Prometheus UI in the browser:

	// Open monitoring page with all metrics.
	if err := mon.OpenUserInterfaceInBrowser(); err != nil {
		return errors.Wrap(err, "open monitoring UI in browser")
	}

To see how it works in practice, run our example code in standalone.go by running make run-example. At the end, four UIs should show in your browser:

  • Thanos one,
  • Monitoring (Prometheus)
  • Profiling (Parca)
  • Tracing (Jaeger).

In the monitoring UI you can then e.g. query docker container metrics using container_memory_working_set_bytes{id!="/"} metric:

mem metric

NOTE: Due to cgroup modifications and using advanced docker features, this might behave different on non-Linux platforms. Let us know in the issue if you encounter any issue on Mac or Windows and help us to add support for those operating systems!

Bonus: Monitoring performance of e2e process itself.

It's common pattern that you want to schedule some containers but also, you might want to monitor a local code you just wrote. For this you can run your local code in an ad-hoc container using e2e.Containerize():

	l, err := e2e.Containerize(e, "run", Run)
	testutil.Ok(t, err)

	testutil.Ok(t, e2e.StartAndWaitReady(l))

While having the Run function in a separate non-test file. The function must be exported, for example:

func Run(ctx context.Context) error {
	// Do something.

	<-ctx.Done()
	return nil
}

This will run your code in a container allowing to use the same monitoring methods thanks to cadvisor.

Continuous Profiling

Similarly to Monitoring, you can wrap your runnable (or instrumented runnable) with e2eprof.AsProfiled if your service uses HTTP pprof handlers (common in Go). When wrapped, you can start continuous profiler using e2eprof package:

prof, err := e2eprof.Start(e)

This will start Parca with automatic discovery for every new and old profiled runnables. Run OpenUserInterfaceInBrowser() to open the Parca UI in the browser:

	// Open profiling page with all profiles.
	if err := prof.OpenUserInterfaceInBrowser(); err != nil {
		return errors.Wrap(err, "open profiling UI in browser")
	}

To see how it works in practice, run our example code in standalone.go by running make run-example. At the end, four UIs should show in your browser:

  • Thanos one,
  • Monitoring (Prometheus)
  • Profiling (Parca)
  • Tracing (Jaeger).

In the profiling UI choose a profile type, filter by instances (autocompleted) and select the profile:

img.png

Monitoring + Profiling

For runnables that are both instrumented and profiled you can use e2eobs.AsObservable.

Troubleshooting
Can't create docker network

If you see an output like the one below:

18:09:11 dockerEnv: [docker ps -a --quiet --filter network=kubelet]
18:09:11 dockerEnv: [docker network ls --quiet --filter name=kubelet]
18:09:11 dockerEnv: [docker network create -d bridge kubelet]
18:09:11 Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

The first potential reasons is that this command often does not work if you have VPN client working like openvpn, expressvpn, nordvpn etc. Unfortunately the fastest solution is to turn off the VPN for the duration of test.

If that is not the reason, consider pruning your docker networks. You might have leftovers from previous runs (although in successful runs, e2e cleans those).

Use docker network prune -f to clean those.

Credits

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildArgs 

func BuildArgs(flags map[string]string) []string

func BuildKingpinArgs added in v0.10.0 

func BuildKingpinArgs(flags map[string]string) []string

BuildKingpinArgs is like BuildArgs but with special handling of slice args. NOTE(bwplotka): flags with values as comma but not indented to be slice will cause issues.

func EmptyFlags 

func EmptyFlags() map[string]string

func MergeFlags 

func MergeFlags(inputs ...map[string]string) map[string]string

func MergeFlagsWithoutRemovingEmpty 

func MergeFlagsWithoutRemovingEmpty(inputs ...map[string]string) map[string]string

func StartAndWaitReady 

func StartAndWaitReady(runnables ...Runnable) error

Types

type CmdReadinessProbe 

type CmdReadinessProbe struct {
	// contains filtered or unexported fields
}

CmdReadinessProbe checks readiness by `Exec`ing a command (within container) which returns 0 to consider status being ready.

func NewCmdReadinessProbe 

func NewCmdReadinessProbe(cmd Command) *CmdReadinessProbe

func (*CmdReadinessProbe) Ready 

func (p *CmdReadinessProbe) Ready(runnable Runnable) error

type Command 

type Command struct {
	Cmd                string
	Args               []string
	EntrypointDisabled bool
}

func NewCommand 

func NewCommand(cmd string, args ...string) Command

func NewCommandRunUntilStop added in v0.12.1 

func NewCommandRunUntilStop() Command

NewCommandRunUntilStop is a command that allows to keep container running.

func NewCommandWithoutEntrypoint 

func NewCommandWithoutEntrypoint(cmd string, args ...string) Command

type DockerEnvironment 

type DockerEnvironment struct {
	// contains filtered or unexported fields
}

DockerEnvironment defines single node docker engine that allows to run Services.

func New added in v0.13.0 

func New(opts ...EnvironmentOption) (_ *DockerEnvironment, err error)

New creates new, isolated docker environment.

func NewDockerEnvironment 

func NewDockerEnvironment(name string, opts ...EnvironmentOption) (_ *DockerEnvironment, err error)

NewDockerEnvironment creates new, isolated docker environment. The `name` option is now deprecated and is equivalent to `e2e.WithName()`. Feel free to leave it empty. Deprecated: Use New instead.

func (*DockerEnvironment) AddCloser added in v0.11.0 

func (e *DockerEnvironment) AddCloser(f func())

func (*DockerEnvironment) AddListener added in v0.10.0 

func (e *DockerEnvironment) AddListener(listener EnvironmentListener)

AddListener registers given listener to be notified on environment runnable changes.

func (*DockerEnvironment) Close 

func (e *DockerEnvironment) Close()

func (*DockerEnvironment) HostAddr added in v0.11.0 

func (e *DockerEnvironment) HostAddr() string

func (*DockerEnvironment) Name added in v0.11.0 

func (e *DockerEnvironment) Name() string

func (*DockerEnvironment) Runnable 

func (e *DockerEnvironment) Runnable(name string) RunnableBuilder

func (*DockerEnvironment) SharedDir 

func (e *DockerEnvironment) SharedDir() string

type Environment 

type Environment interface {
	// Name returns environment name.
	Name() string
	// SharedDir returns host directory that will be shared with all runnables.
	SharedDir() string
	// HostAddr returns host address that is available from runnables.
	HostAddr() string
	// Runnable returns runnable builder which can build runnables that can be started and stopped within this environment.
	Runnable(name string) RunnableBuilder
	// AddListener registers given listener to be notified on environment runnable changes.
	AddListener(listener EnvironmentListener)
	// AddCloser registers function to be invoked on close, before all containers are sent kill signal.
	AddCloser(func())
	// Close shutdowns isolated environment and cleans its resources.
	Close()
}

Environment defines how to run Runnable in isolated area e.g via docker in isolated docker network.

type EnvironmentListener added in v0.10.0 

type EnvironmentListener interface {
	OnRunnableChange(started []Runnable) error
}

type EnvironmentOption 

type EnvironmentOption func(*environmentOptions)

EnvironmentOption defined the signature of a function used to manipulate options.

func WithLogger 

func WithLogger(logger Logger) EnvironmentOption

WithLogger tells environment to use custom logger to default one (stdout).

func WithName added in v0.13.0 

func WithName(name string) EnvironmentOption

WithName injects custom name of environment which will be used as a network name. If the name is not unique across different e2e environments ran at the same moment this will race them. In the same time if name is unique across every environment run for the same test it won't be able to easily clean up (so be reused) on the next run if the network was not cleaned properly.

By default, it creates `e2e_<hash based on function name>` environment name.

NOTE: Some restrictions apply. See https://stackoverflow.com/a/53478768.

func WithVerbose 

func WithVerbose() EnvironmentOption

WithVerbose tells environment to be verbose i.e print all commands it executes.

func WithVolumes added in v0.14.0 

func WithVolumes(volumes ...string) EnvironmentOption

type ExecOption added in v0.12.1 

type ExecOption func(o *ExecOptions)

func WithExecOptionStderr added in v0.12.1 

func WithExecOptionStderr(stderr io.Writer) ExecOption

WithExecOptionStderr sets stderr writer to be used when exec is performed. By default, it is streaming to the env logger.

func WithExecOptionStdout added in v0.12.1 

func WithExecOptionStdout(stdout io.Writer) ExecOption

WithExecOptionStdout sets stdout writer to be used when exec is performed. By default, it is streaming to the env logger.

type ExecOptions added in v0.12.1 

type ExecOptions struct {
	Stdout io.Writer
	Stderr io.Writer
}

type FutureRunnable 

type FutureRunnable interface {
	Linkable

	// Init transforms future into runnable.
	Init(opts StartOptions) Runnable
}

type HTTPReadinessProbe 

type HTTPReadinessProbe struct {
	// contains filtered or unexported fields
}

HTTPReadinessProbe checks readiness by making HTTP or HTTPS call and checking for expected HTTP/HTTPS status code.

func NewHTTPReadinessProbe 

func NewHTTPReadinessProbe(portName string, path string, expectedStatusRangeStart, expectedStatusRangeEnd int, expectedContent ...string) *HTTPReadinessProbe

func NewHTTPSReadinessProbe added in v0.12.0 

func NewHTTPSReadinessProbe(portName, path string, expectedStatusRangeStart, expectedStatusRangeEnd int, expectedContent ...string) *HTTPReadinessProbe

func (*HTTPReadinessProbe) Ready 

func (p *HTTPReadinessProbe) Ready(runnable Runnable) (err error)

type LinePrefixLogger 

type LinePrefixLogger struct {
	// contains filtered or unexported fields
}

func (*LinePrefixLogger) Write 

func (w *LinePrefixLogger) Write(p []byte) (n int, err error)

type Linkable 

type Linkable interface {
	// Name returns unique name for the Runnable instance.
	Name() string

	// Dir returns the working directory path that is shared with the host and this runnable. The paths are exactly
	// the same for runnable and host to enable symbolic links (as long as those don't link to paths outside of Dir or not
	// otherwise shared with runnable).
	Dir() string

	// Deprecated. Use Dir instead. For compatibility Dir() is returned.
	InternalDir() string

	// InternalEndpoint returns internal runnable endpoint (host:port) for given internal port.
	// Internal means that it will be accessible only from runnable context.
	InternalEndpoint(portName string) string
}

Linkable is the entity that one can use to link runnable to other runnables before started.

type Logger 

type Logger interface {
	Log(keyvals ...interface{}) error
}

Logger is the fundamental interface for all log operations. Log creates a log event from keyvals, a variadic sequence of alternating keys and values. Implementations must be safe for concurrent use by multiple goroutines. In particular, any implementation of Logger that appends to keyvals or modifies or retains any of its elements must make a copy first. This is 1:1 copy of "github.com/go-kit/kit/log" interface.

type ReadinessProbe 

type ReadinessProbe interface {
	Ready(runnable Runnable) (err error)
}

type Runnable 

type Runnable interface {
	Linkable
	// contains filtered or unexported methods
}

Runnable is the entity that environment returns to manage single instance.

func Containerize added in v0.12.0 

func Containerize(e Environment, name string, startFn func(context.Context) error) (Runnable, error)

Containerize inspects startFn and builds Go shim with local process endpoint that imports given `startFn` function. Binary is then put in adhoc container and returned as runnable ready to be started.

func NewFailedRunnable added in v0.13.0 

func NewFailedRunnable(name string, err error) Runnable

NewFailedRunnable returns runnable that failed in construction.

type RunnableBuilder added in v0.10.0 

type RunnableBuilder interface {
	// WithPorts adds ports to runnable, allowing caller to
	// use `InternalEndpoint` and `Endpoint` methods by referencing port by name.
	WithPorts(map[string]int) RunnableBuilder
	// Future returns future runnable
	Future() FutureRunnable
	// Init returns runnable.
	Init(opts StartOptions) Runnable
}

RunnableBuilder represents options that can be build into runnable and if you want Future or Initiated Runnable from it.

type RunnableCapabilities added in v0.12.0 

type RunnableCapabilities string
const (
	RunnableCapabilitiesSysAdmin RunnableCapabilities = "SYS_ADMIN"
)

type SimpleLogger 

type SimpleLogger struct {
	// contains filtered or unexported fields
}

func NewLogger 

func NewLogger(w io.Writer) *SimpleLogger

func (*SimpleLogger) Log 

func (l *SimpleLogger) Log(keyvals ...interface{}) error

type StartOptions 

type StartOptions struct {
	Image     string
	EnvVars   map[string]string
	User      string
	Command   Command
	Readiness ReadinessProbe
	// WaitReadyBackofff represents backoff used for WaitReady.
	WaitReadyBackoff *backoff.Config
	Volumes          []string
	UserNs           string
	Privileged       bool
	Capabilities     []RunnableCapabilities

	LimitMemoryBytes uint
	LimitCPUs        float64
}

StartOptions represents starting option of runnable in the environment.

type TCPReadinessProbe 

type TCPReadinessProbe struct {
	// contains filtered or unexported fields
}

TCPReadinessProbe checks readiness by ensure a TCP connection can be established.

func NewTCPReadinessProbe 

func NewTCPReadinessProbe(portName string) *TCPReadinessProbe

func (*TCPReadinessProbe) Ready 

func (p *TCPReadinessProbe) Ready(runnable Runnable) (err error)

Source Files

View all Source files

Directories

Path Synopsis
Package e2edb is a reference, instrumented runnables that are running various popular databases one could run in their tests or benchmarks.
 Package e2edb is a reference, instrumented runnables that are running various popular databases one could run in their tests or benchmarks.
examples
thanos Module
matchers
promconfig
promconfig/discovery/azure
promconfig/discovery/config
promconfig/discovery/consul
promconfig/discovery/dns
promconfig/discovery/ec2
promconfig/discovery/file
promconfig/discovery/gce
promconfig/discovery/kubernetes
promconfig/discovery/marathon
promconfig/discovery/openstack
promconfig/discovery/targetgroup
promconfig/discovery/triton
promconfig/discovery/zookeeper
parcaconfig

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.