framework

README

Istio Test Framework

This document introduces the Istio test framework.

For an overview of the architecture as well as how to extend the framework, see the Developer Guide.

Introduction

Writing tests for cloud-based microservices is hard. Getting the tests to run quickly and reliably is difficult in and of itself. However, supporting multiple cloud platforms is another thing altogether.

The Istio test framework attempts to address these problems. Some of the objectives for the framework are:

• Writing Tests:
  • Platform Agnostic: The API abstracts away the details of the underlying platform. This allows the developer to focus on testing the Istio logic rather than plumbing the infrastructure.
  • Reuseable Tests: Suites of tests can be written which will run against any platform that supports Istio. This effectively makes conformance testing built-in.
• Running Tests:
  • Standard Tools: Built on Go's testing infrastructure and run with standard commands (e.g. go test) .
  • Easy: Few or no flags are required to run tests out of the box. While many flags are available, they are provided reasonable defaults where possible.
  • Fast: With the ability to run processes natively on the host machine, running tests are orders of magnitude faster.
  • Reliable: Running tests natively are inherently more reliable than in-cluster. However, the components for each platform are written to be reliable (e.g. retries).

Getting Started

To begin using the test framework, you'll need a write a TestMain that simply calls framework.Run:

func TestMain(m *testing.M) { 
    framework.Main("my_test", m)
}

The first parameter is a TestID, which can be any string. It's used mainly for creating a working directory for the test output.

The call to framework.Main does the following:

1. Starts the platform-specific environment. By default, the native environment is used. To run on Kubernetes, set the flag: --istio.test.env=kube.
2. Run all tests in the current package. This is the standard Go behavior for TestMain.
3. Stops the environment.

Then, in the same package as TestMain, define your tests:

func TestHTTP(t *testing.T) {
    // Get the test context from the framework.
    ctx := framework.GetContext(t)
    defer ctx.Done(t)
    
    // Get the component(s) that you need.
    apps := components.GetApps(t, ctx)
    a := apps.GetAppOrFail("a", t)
    b := apps.GetAppOrFail("b", t)

    // Interact with the components...
    
    be := b.EndpointsForProtocol(model.ProtocolHTTP)[0]
    result := a.CallOrFail(be, components.AppCallOptions{}, t)[0]
    if !result.IsOK() {
        t.Fatalf("HTTP Request unsuccessful: %s", result.Body)
    }
}

Every test will follow the pattern in the example above:

1. Get the context. The context is the main API for the test framework.
2. Get and use components. Each component (e.g. Pilot, Mixer, Apps) defines its own API. See the interface documentation for details on usage.

If you need to do suite-level checks, then you can pass additional parameters to framework.Main:

func TestMain(m *testing.M) {
    framework.Main("my_test", m,
    framework.RequireEnvironment(environment.Kube), // Require Kubernetes environment.
    istio.SetupOnKube(&ist, setupIstioConfig),            // Deploy Istio, to be used by the whole suite.
    setup)                                   // Call your setup function.
}

func setupIstioConfig(cfg *istio.Config) {
    cfg.Values["your-feature-enabled"] = "true"
}

func setup(ctx core.SuiteContext) error {
  // ...
}

Supported Platforms

Native

Running on the native host platform (i.e. in-process or local processes) is the default. Running natively has several advantages over in-cluster in that they're generally simpler to run, faster, and more reliable.

Running natively requires no flags, since --istio.test.env=native is the default. However, at the time of this writing, not all components have been implemented natively.

Kubernetes

To run on Kubernetes, specify the flag --istio.test.env=kube. By default, Istio will be deployed using the configuration in ~/.kube/config.

Several flags are available to customize the behavior, such as:

Flag	Default	Description
istio.test.env	native	Specify the environment to run the tests against. Allowed values are: kube, native. Defaults to native.
istio.test.work_dir	''	Local working directory for creating logs/temp files. If left empty, os.TempDir() is used.
istio.test.hub	''	Container registry hub to use. If not specified, HUB environment value will be used.
istio.test.tag	''	Common container tag to use when deploying container images. If not specified TAG environment value will be used.
istio.test.pullpolicy	Always	Common image pull policy to use when deploying container images. If not specified PULL_POLICY environment value will be used. Defaults to Always
istio.test.nocleanup	false	Do not cleanup resources after test completion.
istio.test.ci	false	Enable CI Mode. Additional logging and state dumping will be enabled.
istio.test.kube.config	~/.kube/config	Location of the kube config file to be used.
istio.test.kube.minikube	false	If true access to the ingress will be via nodeport. Should be set to true if running on Minikube.
istio.test.kube.systemNamespace	istio-system	Namespace for Istio deployment. If '', the namespace is generated with the prefix "istio-system-".
istio.test.kube.deploy	true	If true, the components should be deployed to the cluster. Otherwise, it is assumed that the components have already deployed.
istio.test.kube.helm.chartDir	$(ISTIO)/install/kubernetes/helm/istio
istio.test.kube.helm.valuesFile	values-e2e.yaml	The name of a file (relative to istio.test.kube.helm.chartDir) to provide Helm values.
istio.test.kube.helm.values	''	A comma-separated list of helm values that will override those provided by istio.test.kube.helm.valuesFile. These are overlaid on top of a map containing the following: global.hub=${HUB}, global.tag=${TAG}, global.proxy.enableCoreDump=true, global.mtls.enabled=true,galley.enabled=true.

Documentation

Index

• func Main(testID string, m *testing.M, setupFn ...resource.SetupFn)
• func Run(t *testing.T, fn func(ctx TestContext))
• type Suite
  • func NewSuite(testID string, m *testing.M) *Suite
  • func (s *Suite) EnvSetup(e environment.Name, fn resource.SetupFn) *Suite
  • func (s *Suite) Label(labels ...label.Instance) *Suite
  • func (s *Suite) RequireEnvironment(name environment.Name) *Suite
  • func (s *Suite) Run()
  • func (s *Suite) Setup(fn resource.SetupFn) *Suite
• type SuiteContext
• type Test
  • func NewTest(t *testing.T) *Test
  • func (t *Test) Label(labels ...label.Instance) *Test
  • func (t *Test) RequiresEnvironment(name environment.Name) *Test
  • func (t *Test) Run(fn func(ctx TestContext))
• type TestContext
  • func NewContext(t *testing.T, labels ...label.Instance) TestContext

Functions

func Main

func Main(testID string, m *testing.M, setupFn ...resource.SetupFn)

Main runs the test suite. The Main will run the supplied setup functions before starting test execution. It will not return, and will exit the process after running tests.

func Run

func Run(t *testing.T, fn func(ctx TestContext))

Run runs the given test.

Types

type Suite

type Suite struct {
	// contains filtered or unexported fields
}

Suite allows the test author to specify suite-related metadata and do setup in a fluent-style, before commencing execution.

func NewSuite

func NewSuite(testID string, m *testing.M) *Suite

NewSuite returns a new suite instance.

func (*Suite) EnvSetup

func (s *Suite) EnvSetup(e environment.Name, fn resource.SetupFn) *Suite

EnvSetup runs the given setup function conditionally, based on the current environment.

func (*Suite) Label

func (s *Suite) Label(labels ...label.Instance) *Suite

Label all the tests in suite with the given labels

func (*Suite) RequireEnvironment

func (s *Suite) RequireEnvironment(name environment.Name) *Suite

RequireEnvironment ensures that the current environment matches what the suite expects. Otherwise it stops test execution. This also applies the appropriate label to the suite implicitly.

func (*Suite) Run

func (s *Suite) Run()

Run the suite. This method calls os.Exit and does not return.

func (*Suite) Setup

func (s *Suite) Setup(fn resource.SetupFn) *Suite

Setup runs enqueues the given setup function to run before test execution.

type SuiteContext

type SuiteContext interface {
	resource.Context
}

SuiteContext contains suite-level items used during runtime.

type Test

type Test struct {
	// contains filtered or unexported fields
}

Test allows the test author to specify test-related metadata in a fluent-style, before commencing execution.

func NewTest

func NewTest(t *testing.T) *Test

NewTest returns a new test wrapper for running a single test.

func (*Test) Label

func (t *Test) Label(labels ...label.Instance) *Test

Label applies the given labels to this test.

func (*Test) RequiresEnvironment

func (t *Test) RequiresEnvironment(name environment.Name) *Test

RequireEnvironment ensures that the current environment matches what the suite expects. Otherwise it stops test execution and skips the test.

func (*Test) Run

func (t *Test) Run(fn func(ctx TestContext))

Run the test, supplied as a lambda.

type TestContext

type TestContext interface {
	resource.Context

	// WorkDir allocated for this test.
	WorkDir() string

	// CreateDirectoryOrFail creates a new sub directory with the given name in the workdir, or fails the test.
	CreateDirectoryOrFail(t *testing.T, name string) string

	// CreateTmpDirectoryOrFail creates a new temporary directory with the given prefix in the workdir, or fails the test.
	CreateTmpDirectoryOrFail(t *testing.T, prefix string) string

	// RequireOrSkip skips the test if the environment is not as expected.
	RequireOrSkip(t *testing.T, envName environment.Name)

	// Done should be called when this context is to be cleaned up
	Done(t *testing.T)
}

TestContext is a test-level context that can be created as part of test executing tests.

func NewContext

func NewContext(t *testing.T, labels ...label.Instance) TestContext

NewContext creates a new test context and returns. It is upto the caller to close to context by calling .Done() at the end of the test run.