venom

README

Venom

Venom run executors (script, HTTP Request, etc... ) and assertions. It can also output xUnit results files.

Commmand Line

Install with:

$ go get github.com/ovh/venom
$ go install github.com/ovh/venom/cli/venom

$ venom run -h
Run Tests

Usage:
  venom run [flags]

Flags:
      --details string        Output Details Level : low, medium, high (default "medium")
      --env                   Inject environment variables. export FOO=BAR -> you can use {{.FOO}} in your tests (default true)
      --exclude stringSlice   --exclude filaA.yaml --exclude filaB.yaml --exclude fileC*.yaml
      --format string         --format:yaml, json, xml, tap (default "xml")
      --log string            Log Level : debug, info or warn (default "warn")
      --output-dir string     Output Directory: create tests results file inside this directory
      --parallel int          --parallel=2 : launches 2 Test Suites in parallel (default 1)
      --resume                Output Resume: one line with Total, TotalOK, TotalKO, TotalSkipped, TotalTestSuite (default true)
      --resumeFailures        Output Resume Failures (default true)
      --var stringSlice       --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file         --var-from-file filename.yaml : yaml|json, must contains map[string]string

TestSuite files

• Run venom template
• Examples: https://github.com/ovh/cds/tree/master/tests

Example:

name: Title of TestSuite
testcases:
- name: TestCase with default value, exec cmd. Check if exit code != 1
  steps:
  - script: echo 'foo'
    type: exec

- name: Title of First TestCase
  steps:
  - script: echo 'foo'
    assertions:
    - result.code ShouldEqual 0
  - script: echo 'bar'
    assertions:
    - result.systemout ShouldNotContainSubstring foo
    - result.timeseconds ShouldBeLessThan 1

- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    timeout: 5
    assertions:
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1

- name: Test with retries and delay in seconds between each try
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    retry: 3
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200

Using variables and reuse results

name: MyTestSuiteTmpl
vars:
  api.foo: 'http://api/foo'
  second: 'venomWithTmpl'

testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.api.foo}}'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual http://api/foo

- name: testB
  steps:
  - type: exec
    script: echo 'XXX{{.testA.result.systemout}}YYY'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual XXXhttp://api/fooYYY

Extract variable from results and reuse it in step after

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo 'foo with a bar here'
    extracts:
      result.systemout: foo with a {{myvariable=[a-z]+}} here

- name: testB
  steps:
  - type: exec
    script: echo {{.testA.myvariable}}
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldContainSubstring bar

RUN Venom locally on CDS Integration Tests

cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var cdsro='cds -f $HOME/.cds/it.user.ro.json' --var cds='cds -f $HOME/.cds/it.user.rw.json' --parallel=5

RUN Venom with file var

vars.yaml :

cdsro: 'cds -f $HOME/.cds/it.user.ro.json'
cds: 'cds -f $HOME/.cds/it.user.rw.json'

cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var-from-file vars.yaml --parallel=5

RUN Venom, with an export xUnit

venom run  --details=low --format=xml --output-dir="."

Executors

• exec: https://github.com/ovh/venom/tree/master/executors/exec exec is the default type for a step
• http: https://github.com/ovh/venom/tree/master/executors/http
• imap: https://github.com/ovh/venom/tree/master/executors/imap
• readfile: https://github.com/ovh/venom/tree/master/executors/readfile
• smtp: https://github.com/ovh/venom/tree/master/executors/smtp
• ssh: https://github.com/ovh/venom/tree/master/executors/ssh
• web: https://github.com/ovh/venom/tree/master/executors/web

Assertion

Keywords

• ShouldEqual
• ShouldNotEqual
• ShouldAlmostEqual
• ShouldNotAlmostEqual
• ShouldResemble
• ShouldNotResemble
• ShouldPointTo
• ShouldNotPointTo
• ShouldBeNil
• ShouldNotBeNil
• ShouldBeTrue
• ShouldBeFalse
• ShouldBeZeroValue
• ShouldBeGreaterThan
• ShouldBeGreaterThanOrEqualTo
• ShouldBeLessThan
• ShouldBeLessThanOrEqualTo
• ShouldBeBetween
• ShouldNotBeBetween
• ShouldBeBetweenOrEqual
• ShouldNotBeBetweenOrEqual
• ShouldContain
• ShouldNotContain
• ShouldContainKey
• ShouldNotContainKey
• ShouldBeIn
• ShouldNotBeIn
• ShouldBeEmpty
• ShouldNotBeEmpty
• ShouldHaveLength
• ShouldStartWith
• ShouldNotStartWith
• ShouldEndWith
• ShouldNotEndWith
• ShouldBeBlank
• ShouldNotBeBlank
• ShouldContainSubstring
• ShouldNotContainSubstring
• ShouldEqualWithout
• ShouldEqualTrimSpace
• ShouldHappenBefore
• ShouldHappenOnOrBefore
• ShouldHappenAfter
• ShouldHappenOnOrAfter
• ShouldHappenBetween
• ShouldHappenOnOrBetween
• ShouldNotHappenOnOrBetween
• ShouldHappenWithin
• ShouldNotHappenWithin
• ShouldBeChronological

Write your executor

An executor have to implement this interface

// Executor execute a testStep.
type Executor interface {
	// Run run a Test Step
	Run(ctx context.Content, venom.Logger, TestStep) (ExecutorResult, error)
}

Example

// Name of executor
const Name = "myexecutor"

// New returns a new Executor
func New() venom.Executor {
	return &Executor{}
}

// Executor struct
type Executor struct {
	Command string `json:"command,omitempty" yaml:"command,omitempty"`
}

// Result represents a step result
type Result struct {
	Code        int    `json:"code,omitempty" yaml:"code,omitempty"`
	Command     string `json:"command,omitempty" yaml:"command,omitempty"`
	Systemout   string   `json:"systemout,omitempty" yaml:"systemout,omitempty"` // put in testcase.Systemout by venom if present
	Systemerr   string   `json:"systemerr,omitempty" yaml:"systemerr,omitempty"` // put in testcase.Systemerr by venom if present
	Executor    Executor `json:"executor,omitempty" yaml:"executor,omitempty"`  
}

// GetDefaultAssertions return default assertions for this executor
// Optional
func (Executor) GetDefaultAssertions() venom.StepAssertions {
	return venom.StepAssertions{Assertions: []string{"result.code ShouldEqual 0"}}
}

// Run execute TestStep
func (Executor) Run(ctx context.Context, l venom.Logger, step venom.TestStep) (venom.ExecutorResult, error) {

	// transform step to Executor Instance
	var t Executor
	if err := mapstructure.Decode(step, &t); err != nil {
		return nil, err
	}

	// Get testcase context if needed
	varContext := ctx.Value(venom.ContextKey).(map[string]interface{})
	if varContext == nil {
        return nil, fmt.Errorf("Executor web need a context")
    }
    bar := varContext['foo']

	// to something with t.Command here...
	//...

	systemout := "foo"
	ouputCode := 0

	// prepare result
	r := Result{
		Code:    ouputCode, // return Output Code
		Command: t.Command, // return Command executed
		Systemout:  systemout,    // return Output string
		Executor: t, // return executor, usefull for display Executor context in failure
	}

	return dump.ToMap(r)
}

Feel free to open a Pull Request with your executors.

TestCase Context

TestCase Context allows you to inject datas in all Steps.

Define a context is optional, but can be usefull to keep data between teststeps on a testcase.

Write your TestCase Context

A TestCase Context has to implement this interface

type TestCaseContext interface {
	Init() error
	Close() error
	SetTestCase(tc TestCase)
	GetName() string
}

Example

// Context Type name
const Name = "default"

// New returns a new TestCaseContext
func New() venom.TestCaseContext {
	ctx := &DefaultTestCaseContext{}
	ctx.Name = Name
	return ctx
}

// DefaultTestCaseContext represents the context of a testcase
type DefaultTestCaseContext struct {
	venom.CommonTestCaseContext
	datas map[string]interface{}
}

// Init Initialize the context
func (tcc *DefaultTestCaseContext) Init() error {
	return nil
}

// Close the context
func (tcc *DefaultTestCaseContext) Close() error {
	return nil
}

Methods SetTestCase and GetName are implemented by CommonTestCaseContext

Hacking

You've developed a new cool feature? Fixed an annoying bug? We'd be happy to hear from you! Make sure to read CONTRIBUTING.md before.

License

This work is under the BSD license, see the LICENSE file for details.

Documentation

Index

• Constants
• Variables
• func Exit(format string, args ...interface{})
• func OutputResult(format string, resume, resumeFailures bool, outputDir string, tests Tests, ...) error
• func RegisterExecutor(name string, e Executor)
• func RegisterTestCaseContext(name string, tcc TestCaseContext)
• func ShouldContainSubstring(actual interface{}, expected ...interface{}) string
• type Aliases
• type CommonTestCaseContext
  • func (t *CommonTestCaseContext) GetName() string
  • func (t *CommonTestCaseContext) SetTestCase(tc TestCase)
• type Executor
• type ExecutorResult
  • func RunTestStep(tcc TestCaseContext, e *ExecutorWrap, ts *TestSuite, tc *TestCase, ...) ExecutorResult
• type ExecutorWrap
  • func WrapExecutor(t map[string]interface{}, tcc TestCaseContext) (*ExecutorWrap, error)
• type Failure
• type InnerResult
• type Logger
• type Property
• type StepAssertions
• type StepExtracts
• type Templater
  • func (tmpl *Templater) Add(prefix string, values map[string]string)
  • func (tmpl *Templater) ApplyOnContext(ctx map[string]interface{}) (map[string]interface{}, error)
  • func (tmpl *Templater) ApplyOnStep(step TestStep) (TestStep, error)
• type TestCase
• type TestCaseContext
  • func ContextWrap(tc *TestCase) (TestCaseContext, error)
• type TestStep
• type TestSuite
• type Tests
  • func Process(path []string, variables map[string]string, exclude []string, parallel int, ...) (*Tests, error)

Constants

const (
	// DetailsLow prints only summary results
	DetailsLow = "low"
	// DetailsMedium prints progress bar and summary
	DetailsMedium = "medium"
	// DetailsHigh prints progress bar and details
	DetailsHigh = "high"
)

const (
	// ContextKey is key for Test Case Context. this
	// can be used by executors for getting context
	ContextKey = "tcContext"
)

const Version = "0.14.0"

Version of Venom One Line for this, used by release.sh script Keep "const Version on one line"

Variables

var PrintFunc = fmt.Printf

PrintFunc used by venom to print output

Functions

func Exit

func Exit(format string, args ...interface{})

Exit func display an error message on stderr and exit 1

func OutputResult

func OutputResult(format string, resume, resumeFailures bool, outputDir string, tests Tests, elapsed time.Duration, detailsLevel string) error

OutputResult output result to sdtout, files...

func RegisterExecutor

func RegisterExecutor(name string, e Executor)

RegisterExecutor register Test Executors

func RegisterTestCaseContext

func RegisterTestCaseContext(name string, tcc TestCaseContext)

RegisterTestCaseContext new register TestCaseContext

func ShouldContainSubstring

func ShouldContainSubstring(actual interface{}, expected ...interface{}) string

ShouldContainSubstring receives exactly more than 2 string parameters and ensures that the first contains the second as a substring.

Types

type Aliases

type Aliases map[string]string

Aliases contains list of aliases

type CommonTestCaseContext

type CommonTestCaseContext struct {
	TestCaseContext
	TestCase TestCase
	Name     string
}

CommonTestCaseContext represents a Default TestCase Context

func (*CommonTestCaseContext) GetName

func (t *CommonTestCaseContext) GetName() string

GetName Get the context name

func (*CommonTestCaseContext) SetTestCase

func (t *CommonTestCaseContext) SetTestCase(tc TestCase)

SetTestCase set testcase in context

type Executor

type Executor interface {
	// Run run a Test Step
	Run(TestCaseContext, Logger, TestStep) (ExecutorResult, error)
}

Executor execute a testStep.

type ExecutorResult

type ExecutorResult map[string]string

ExecutorResult represents an executor result on a test step

func RunTestStep

func RunTestStep(tcc TestCaseContext, e *ExecutorWrap, ts *TestSuite, tc *TestCase, step TestStep, templater *Templater, l Logger, detailsLevel string) ExecutorResult

RunTestStep executes a venom testcase is a venom context

type ExecutorWrap

type ExecutorWrap struct {
	// contains filtered or unexported fields
}

ExecutorWrap contains an executor implementation and some attributes

func WrapExecutor

func WrapExecutor(t map[string]interface{}, tcc TestCaseContext) (*ExecutorWrap, error)

WrapExecutor initializes a test by name no type -> exec is default

type Failure

type Failure struct {
	Value   string `xml:",cdata" json:"value" yaml:"value,omitempty"`
	Type    string `xml:"type,attr,omitempty" json:"type" yaml:"type,omitempty"`
	Message string `xml:"message,attr,omitempty" json:"message" yaml:"message,omitempty"`
}

Failure contains data related to a failed test.

type InnerResult

type InnerResult struct {
	Value string `xml:",cdata" json:"value" yaml:"value"`
}

InnerResult is used by TestCase

type Logger

type Logger interface {
	Debugf(format string, args ...interface{})
	Infof(format string, args ...interface{})
	Warnf(format string, args ...interface{})
	Warningf(format string, args ...interface{})
	Errorf(format string, args ...interface{})
	Fatalf(format string, args ...interface{})
}

Logger is basicaly an interface for logrus.Entry

type Property

type Property struct {
	XMLName xml.Name `xml:"property" json:"-" yaml:"-"`
	Name    string   `xml:"name,attr" json:"name" yaml:"-"`
	Value   string   `xml:"value,attr" json:"value" yaml:"-"`
}

Property represents a key/value pair used to define properties.

type StepAssertions

type StepAssertions struct {
	Assertions []string `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}

StepAssertions contains step assertions

type StepExtracts

type StepExtracts struct {
	Extracts map[string]string `json:"extracts,omitempty" yaml:"extracts,omitempty"`
}

StepExtracts contains "step extracts"

type Templater

type Templater struct {
	Values map[string]string
}

Templater contains templating values on a testsuite

func (*Templater) Add

func (tmpl *Templater) Add(prefix string, values map[string]string)

Add add data to templater

func (*Templater) ApplyOnContext

func (tmpl *Templater) ApplyOnContext(ctx map[string]interface{}) (map[string]interface{}, error)

ApplyOnContext executes the template on a context

func (*Templater) ApplyOnStep

func (tmpl *Templater) ApplyOnStep(step TestStep) (TestStep, error)

ApplyOnStep executes the template on a test step

type TestCase

type TestCase struct {
	XMLName    xml.Name               `xml:"testcase" json:"-" yaml:"-"`
	Assertions string                 `xml:"assertions,attr,omitempty" json:"assertions" yaml:"-"`
	Classname  string                 `xml:"classname,attr,omitempty" json:"classname" yaml:"-"`
	Errors     []Failure              `xml:"error,omitempty" json:"errors" yaml:"errors,omitempty"`
	Failures   []Failure              `xml:"failure,omitempty" json:"failures" yaml:"failures,omitempty"`
	Name       string                 `xml:"name,attr" json:"name" yaml:"name"`
	Skipped    int                    `xml:"skipped,attr,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Status     string                 `xml:"status,attr,omitempty" json:"status" yaml:"status,omitempty"`
	Systemout  InnerResult            `xml:"system-out,omitempty" json:"systemout" yaml:"systemout,omitempty"`
	Systemerr  InnerResult            `xml:"system-err,omitempty" json:"systemerr" yaml:"systemerr,omitempty"`
	Time       string                 `xml:"time,attr,omitempty" json:"time" yaml:"time,omitempty"`
	TestSteps  []TestStep             `xml:"-" json:"steps" yaml:"steps"`
	Context    map[string]interface{} `xml:"-" json:"-" yaml:"context,omitempty"`
}

TestCase is a single test case with its result.

type TestCaseContext

type TestCaseContext interface {
	Init() error
	Close() error
	SetTestCase(tc TestCase)
	GetName() string
}

TestCaseContext represents the context of a testcase

func ContextWrap

func ContextWrap(tc *TestCase) (TestCaseContext, error)

ContextWrap initializes a context for a testcase no type -> parent context

type TestStep

type TestStep map[string]interface{}

TestStep represents a testStep

type TestSuite

type TestSuite struct {
	XMLName    xml.Name               `xml:"testsuite" json:"-" yaml:"-"`
	Disabled   int                    `xml:"disabled,attr,omitempty" json:"disabled" yaml:"-"`
	Errors     int                    `xml:"errors,attr,omitempty" json:"errors" yaml:"-"`
	Failures   int                    `xml:"failures,attr,omitempty" json:"failures" yaml:"-"`
	Hostname   string                 `xml:"hostname,attr,omitempty" json:"hostname" yaml:"-"`
	ID         string                 `xml:"id,attr,omitempty" json:"id" yaml:"-"`
	Name       string                 `xml:"name,attr" json:"name" yaml:"name"`
	Package    string                 `xml:"package,attr,omitempty" json:"package" yaml:"-"`
	Properties []Property             `xml:"-" json:"properties" yaml:"-"`
	Skipped    int                    `xml:"skipped,attr,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Total      int                    `xml:"tests,attr" json:"total" yaml:"total,omitempty"`
	TestCases  []TestCase             `xml:"testcase" json:"tests" yaml:"testcases"`
	Time       string                 `xml:"time,attr,omitempty" json:"time" yaml:"-"`
	Timestamp  string                 `xml:"timestamp,attr,omitempty" json:"timestamp" yaml:"-"`
	Vars       map[string]interface{} `xml:"-" json:"-" yaml:"vars"`
	Templater  *Templater             `xml:"-" json:"-" yaml:"-"`
}

TestSuite is a single JUnit test suite which may contain many testcases.

type Tests

type Tests struct {
	XMLName      xml.Name    `xml:"testsuites" json:"-" yaml:"-"`
	Total        int         `xml:"-" json:"total"`
	TotalOK      int         `xml:"-" json:"ok"`
	TotalKO      int         `xml:"-" json:"ko"`
	TotalSkipped int         `xml:"-" json:"skipped"`
	TestSuites   []TestSuite `xml:"testsuite" json:"test_suites"`
}

Tests contains all informations about tests in a pipeline build

func Process

func Process(path []string, variables map[string]string, exclude []string, parallel int, logLevel string, detailsLevel string, writer io.Writer) (*Tests, error)

Process runs tests suite and return a Tests result