dockertest

README

Use Docker to run your Golang integration tests against third party services on Microsoft Windows, Mac OSX and Linux!

Table of Contents

• Why should I use Dockertest?
• Installing and using Dockertest
  • Using Dockertest
  • Examples
• Troubleshoot & FAQ
  • Out of disk space
  • Removing old containers

Why should I use Dockertest?

When developing applications, it is often necessary to use services that talk to a database system. Unit Testing these services can be cumbersome because mocking database/DBAL is strenuous. Making slight changes to the schema implies rewriting at least some, if not all of the mocks. The same goes for API changes in the DBAL. To avoid this, it is smarter to test these specific services against a real database that is destroyed after testing. Docker is the perfect system for running unit tests as you can spin up containers in a few seconds and kill them when the test completes. The Dockertest library provides easy to use commands for spinning up Docker containers and using them for your tests.

Installing and using Dockertest

Using Dockertest is straightforward and simple. Check the releases tab for available releases.

To install dockertest, run

go get -u github.com/ory/dockertest/v3

or

dep ensure -add github.com/ory/dockertest@v3.x.y

Using Dockertest

package dockertest_test

import (
	"database/sql"
	"fmt"
	"log"
	"os"
	"testing"

	_ "github.com/go-sql-driver/mysql"
	"github.com/ory/dockertest/v3"
)

var db *sql.DB

func TestMain(m *testing.M) {
	// uses a sensible default on windows (tcp/http) and linux/osx (socket)
	pool, err := dockertest.NewPool("")
	if err != nil {
		log.Fatalf("Could not construct pool: %s", err)
	}

	// uses pool to try to connect to Docker
	err = pool.Client.Ping()
	if err != nil {
		log.Fatalf("Could not connect to Docker: %s", err)
	}

	// pulls an image, creates a container based on it and runs it
	resource, err := pool.Run("mysql", "5.7", []string{"MYSQL_ROOT_PASSWORD=secret"})
	if err != nil {
		log.Fatalf("Could not start resource: %s", err)
	}

	// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
	if err := pool.Retry(func() error {
		var err error
		db, err = sql.Open("mysql", fmt.Sprintf("root:secret@(localhost:%s)/mysql", resource.GetPort("3306/tcp")))
		if err != nil {
			return err
		}
		return db.Ping()
	}); err != nil {
		log.Fatalf("Could not connect to database: %s", err)
	}

	code := m.Run()

	// You can't defer this because os.Exit doesn't care for defer
	if err := pool.Purge(resource); err != nil {
		log.Fatalf("Could not purge resource: %s", err)
	}

	os.Exit(code)
}

func TestSomething(t *testing.T) {
	// db.Query()
}

Examples

We provide code examples for well known services in the examples directory, check them out!

Troubleshoot & FAQ

Out of disk space

Try cleaning up the images with docker-cleanup-volumes.

Removing old containers

Sometimes container clean up fails. Check out this stackoverflow question on how to fix this. You may also set an absolute lifetime on containers:

resource.Expire(60) // Tell docker to hard kill the container in 60 seconds

To let stopped containers removed from file system automatically, use pool.RunWithOptions() instead of pool.Run() with config.AutoRemove set to true, e.g.:

postgres, err := pool.RunWithOptions(&dockertest.RunOptions{
	Repository: "postgres",
	Tag:        "11",
	Env: []string{
		"POSTGRES_USER=test",
		"POSTGRES_PASSWORD=test",
		"listen_addresses = '*'",
	},
}, func(config *docker.HostConfig) {
	// set AutoRemove to true so that stopped container goes away by itself
	config.AutoRemove = true
	config.RestartPolicy = docker.RestartPolicy{
		Name: "no",
	}
})

Running dockertest in Gitlab CI

How to run dockertest on shared gitlab runners?

You should add docker dind service to your job which starts in sibling container. That means database will be available on host docker.
You app should be able to change db host through environment variable.

Here is the simple example of gitlab-ci.yml:

stages:
  - test
go-test:
  stage: test
  image: golang:1.15
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_DRIVER: overlay2
    YOUR_APP_DB_HOST: docker
  script:
    - go test ./...

Plus in the pool.Retry method that checks for connection readiness, you need to use $YOUR_APP_DB_HOST instead of localhost.

How to run dockertest on group(custom) gitlab runners?

Gitlab runner can be run in docker executor mode to save compatibility with shared runners.
Here is the simple register command:

gitlab-runner register -n \
 --url https://gitlab.com/ \
 --registration-token $YOUR_TOKEN \
 --executor docker \
 --description "My Docker Runner" \
 --docker-image "docker:19.03.12" \
 --docker-privileged

You only need to instruct docker dind to start with disabled tls.
Add variable DOCKER_TLS_CERTDIR: "" to gitlab-ci.yml above. It will tell docker daemon to start on 2375 port over http.

Running Dockertest Using GitHub Actions

name: Test with Docker

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      dind:
        image: docker:23.0-rc-dind-rootless
        ports:
          - 2375:2375
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: "1.18"

      - name: Test with Docker
        run: go test -v ./...

How to run dockertest with remote Docker

Use-case: locally installed docker CLI (client), docker daemon somewhere remotely, environment properly set (ie: DOCKER_HOST, etc..). For example, remote docker can be provisioned by docker-machine.

Currently, dockertest in case of resource.GetHostPort() will return docker host binding address (commonly - localhost) instead of remote docker host. Universal solution is:

func getHostPort(resource *dockertest.Resource, id string) string {
	dockerURL := os.Getenv("DOCKER_HOST")
	if dockerURL == "" {
		return resource.GetHostPort(id)
	}
	u, err := url.Parse(dockerURL)
	if err != nil {
		panic(err)
	}
	return u.Hostname() + ":" + resource.GetPort(id)
}

It will return the remote docker host concatenated with the allocated port in case DOCKER_HOST env is defined. Otherwise, it will fall back to the embedded behavior.

Documentation

Index

• Variables
• type BuildOptions
• type ExecOptions
• type Network
  • func (n *Network) Close() error
• type Pool
  • func NewPool(endpoint string) (*Pool, error)
  • func NewTLSPool(endpoint, certpath string) (*Pool, error)
  • func (d *Pool) BuildAndRun(name, dockerfilePath string, env []string) (*Resource, error)
  • func (d *Pool) BuildAndRunWithBuildOptions(buildOpts *BuildOptions, runOpts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)
  • func (d *Pool) BuildAndRunWithOptions(dockerfilePath string, opts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)
  • func (d *Pool) ContainerByName(containerName string) (*Resource, bool)
  • func (d *Pool) CreateNetwork(name string, opts ...func(config *dc.CreateNetworkOptions)) (*Network, error)
  • func (d *Pool) CurrentContainer() (*Resource, error)
  • func (d *Pool) NetworksByName(name string) ([]Network, error)
  • func (d *Pool) Purge(r *Resource) error
  • func (d *Pool) RemoveContainerByName(containerName string) error
  • func (d *Pool) RemoveNetwork(network *Network) error
  • func (d *Pool) Retry(op func() error) error
  • func (d *Pool) Run(repository, tag string, env []string) (*Resource, error)
  • func (d *Pool) RunWithOptions(opts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)
• type Resource
  • func (r *Resource) Close() error
  • func (r *Resource) ConnectToNetwork(network *Network) error
  • func (r *Resource) DisconnectFromNetwork(network *Network) error
  • func (r *Resource) Exec(cmd []string, opts ExecOptions) (exitCode int, err error)
  • func (r *Resource) Expire(seconds uint) error
  • func (r *Resource) GetBoundIP(id string) string
  • func (r *Resource) GetHostPort(portID string) string
  • func (r *Resource) GetIPInNetwork(network *Network) string
  • func (r *Resource) GetPort(id string) string
• type RunOptions

Variables

var (
	ErrNotInContainer = errors.New("not running in container")
)

Types

type BuildOptions

type BuildOptions struct {
	Dockerfile string
	ContextDir string
	BuildArgs  []dc.BuildArg
	Platform   string
}

BuildOptions is used to pass in optional parameters when building a container

type ExecOptions

type ExecOptions struct {
	// Command environment, optional.
	Env []string

	// StdIn will be attached as command stdin if provided.
	StdIn io.Reader

	// StdOut will be attached as command stdout if provided.
	StdOut io.Writer

	// StdErr will be attached as command stdout if provided.
	StdErr io.Writer

	// Allocate TTY for command or not.
	TTY bool
}

type Network

type Network struct {
	Network *dc.Network
	// contains filtered or unexported fields
}

Network represents a docker network.

func (*Network) Close

func (n *Network) Close() error

Close removes network by calling pool.RemoveNetwork.

type Pool

type Pool struct {
	Client  *dc.Client
	MaxWait time.Duration
}

Pool represents a connection to the docker API and is used to create and remove docker images.

func NewPool

func NewPool(endpoint string) (*Pool, error)

NewPool creates a new pool. You can pass an empty string to use the default, which is taken from the environment variable DOCKER_HOST and DOCKER_URL, or from docker-machine if the environment variable DOCKER_MACHINE_NAME is set, or if neither is defined a sensible default for the operating system you are on. TLS pools are automatically configured if the DOCKER_CERT_PATH environment variable exists.

func NewTLSPool

func NewTLSPool(endpoint, certpath string) (*Pool, error)

NewTLSPool creates a new pool given an endpoint and the certificate path. This is required for endpoints that require TLS communication.

func (*Pool) BuildAndRun

func (d *Pool) BuildAndRun(name, dockerfilePath string, env []string) (*Resource, error)

BuildAndRun builds and starts a docker container

func (*Pool) BuildAndRunWithBuildOptions

func (d *Pool) BuildAndRunWithBuildOptions(buildOpts *BuildOptions, runOpts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)

BuildAndRunWithBuildOptions builds and starts a docker container. Optional modifier functions can be passed in order to change the hostconfig values not covered in RunOptions

func (*Pool) BuildAndRunWithOptions

func (d *Pool) BuildAndRunWithOptions(dockerfilePath string, opts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)

BuildAndRunWithOptions builds and starts a docker container. Optional modifier functions can be passed in order to change the hostconfig values not covered in RunOptions

func (*Pool) ContainerByName

func (d *Pool) ContainerByName(containerName string) (*Resource, bool)

ContainerByName finds a container with the given name and returns it if present

func (*Pool) CreateNetwork

func (d *Pool) CreateNetwork(name string, opts ...func(config *dc.CreateNetworkOptions)) (*Network, error)

CreateNetwork creates docker network. It's useful for linking multiple containers.

func (*Pool) CurrentContainer

func (d *Pool) CurrentContainer() (*Resource, error)

CurrentContainer returns current container descriptor if this function called within running container. It returns ErrNotInContainer as error if this function running not in container.

func (*Pool) NetworksByName

func (d *Pool) NetworksByName(name string) ([]Network, error)

NetworksByName returns a list of docker networks filtered by name

func (*Pool) Purge

func (d *Pool) Purge(r *Resource) error

Purge removes a container and linked volumes from docker.

func (*Pool) RemoveContainerByName

func (d *Pool) RemoveContainerByName(containerName string) error

RemoveContainerByName find a container with the given name and removes it if present

func (*Pool) RemoveNetwork

func (d *Pool) RemoveNetwork(network *Network) error

RemoveNetwork disconnects containers and removes provided network.

func (*Pool) Retry

func (d *Pool) Retry(op func() error) error

Retry is an exponential backoff retry helper. You can use it to wait for e.g. mysql to boot up.

func (*Pool) Run

func (d *Pool) Run(repository, tag string, env []string) (*Resource, error)

Run starts a docker container.

pool.Run("mysql", "5.3", []string{"FOO=BAR", "BAR=BAZ"})

func (*Pool) RunWithOptions

func (d *Pool) RunWithOptions(opts *RunOptions, hcOpts ...func(*dc.HostConfig)) (*Resource, error)

RunWithOptions starts a docker container. Optional modifier functions can be passed in order to change the hostconfig values not covered in RunOptions

 pool.RunWithOptions(&RunOptions{Repository: "mongo", Cmd: []string{"mongod", "--smallfiles"}})
 pool.RunWithOptions(&RunOptions{Repository: "mongo", Cmd: []string{"mongod", "--smallfiles"}}, func(hostConfig *dc.HostConfig) {
			hostConfig.ShmSize = shmemsize
		})

type Resource

type Resource struct {
	Container *dc.Container
	// contains filtered or unexported fields
}

Resource represents a docker container.

func (*Resource) Close

func (r *Resource) Close() error

Close removes a container and linked volumes from docker by calling pool.Purge.

func (*Resource) ConnectToNetwork

func (r *Resource) ConnectToNetwork(network *Network) error

ConnectToNetwork connects container to network.

func (*Resource) DisconnectFromNetwork

func (r *Resource) DisconnectFromNetwork(network *Network) error

DisconnectFromNetwork disconnects container from network.

func (*Resource) Exec

func (r *Resource) Exec(cmd []string, opts ExecOptions) (exitCode int, err error)

Exec executes command within container.

func (*Resource) Expire

func (r *Resource) Expire(seconds uint) error

Expire sets a resource's associated container to terminate after a period has passed

func (*Resource) GetBoundIP

func (r *Resource) GetBoundIP(id string) string

GetBoundIP returns a resource's published IP address.

func (*Resource) GetHostPort

func (r *Resource) GetHostPort(portID string) string

GetHostPort returns a resource's published port with an address.

func (*Resource) GetIPInNetwork

func (r *Resource) GetIPInNetwork(network *Network) string

GetIPInNetwork returns container IP address in network.

func (*Resource) GetPort

func (r *Resource) GetPort(id string) string

GetPort returns a resource's published port. You can use it to connect to the service via localhost, e.g. tcp://localhost:1231/

type RunOptions

type RunOptions struct {
	Hostname     string
	Name         string
	Repository   string
	Tag          string
	Env          []string
	Entrypoint   []string
	Cmd          []string
	Mounts       []string
	Links        []string
	ExposedPorts []string
	ExtraHosts   []string
	CapAdd       []string
	SecurityOpt  []string
	DNS          []string
	WorkingDir   string
	NetworkID    string
	Networks     []*Network // optional networks to join
	Labels       map[string]string
	Auth         dc.AuthConfiguration
	PortBindings map[dc.Port][]dc.PortBinding
	Privileged   bool
	User         string
	Tty          bool
	Platform     string
}

RunOptions is used to pass in optional parameters when running a container.