venom

package module
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2020 License: BSD-3-Clause Imports: 30 Imported by: 38

README

🐍 Venom

Venom run executors (script, HTTP Request, etc. ) and assertions. It can also output xUnit results files.

Venom Demonstration

Command Line

Download latest binary release from https://github.com/ovh/venom/releases
or just go get it go get -u github.com/ovh/venom/cli/venom

$ venom run -h
Run Tests

Usage:
  venom run [flags]

Flags:
      --env                    Inject environment variables. export FOO=BAR -> you can use {{.FOO}} in your tests (default true)
      --exclude strings        --exclude filaA.yaml --exclude filaB.yaml --exclude fileC*.yaml
      --format string          --format:yaml, json, xml, tap (default "xml")
  -h, --help                   help for run
      --log string             Log Level : debug, info or warn (default "warn")
      --no-check-variables     Don't check variables before run
      --output-dir string      Output Directory: create tests results file inside this directory
      --parallel int           --parallel=2 : launches 2 Test Suites in parallel (default 1)
      --profiling              Enable Mem / CPU Profile with pprof
      --stop-on-failure        Stop running Test Suite on first Test Case failure
      --strict                 Exit with an error code if one test fails
      --var strings            --var cds='cds -f config.json' --var cds2='cds -f config.json'
      --var-from-file strings  --var-from-file filename.yaml --var-from-file filename2.yaml : hcl|json|yaml, must contains map[string]string'

Executors

TestSuite files

Example:

name: Title of TestSuite
testcases:
- name: TestCase with default value, exec cmd. Check if exit code != 1
  steps:
  - script: echo 'foo'
    type: exec

- name: Title of First TestCase
  steps:
  - script: echo 'foo'
    assertions:
    - result.code ShouldEqual 0
  - script: echo 'bar'
    assertions:
    - result.systemout ShouldNotContainSubstring foo
    - result.timeseconds ShouldBeLessThan 1

- name: GET http testcase, with 5 seconds timeout
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    timeout: 5
    assertions:
    - result.body ShouldContainSubstring /dedicated/server
    - result.body ShouldContainSubstring /ipLoadbalancing
    - result.statuscode ShouldEqual 200
    - result.timeseconds ShouldBeLessThan 1

- name: Test with retries and delay in seconds between each try
  steps:
  - type: http
    method: GET
    url: https://eu.api.ovh.com/1.0/
    retry: 3
    delay: 2
    assertions:
    - result.statuscode ShouldEqual 200

Using variables and reuse results

name: MyTestSuiteTmpl
vars:
  api.foo: 'http://api/foo'
  second: 'venomWithTmpl'

testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.api.foo}}'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual http://api/foo

- name: testB
  steps:
  - type: exec
    script: echo 'XXX{{.testA.result.systemout}}YYY'
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldEqual XXXhttp://api/fooYYY

Extract variable on the fly from results and reuse it in step after

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo 'foo with a bar here'
    vars:
      myvariable:
        from: result.systemout
        regex: foo with a ([a-z]+) here

- name: testB
  steps:
  - type: exec
    script: echo {{.testA.myvariable}}
    assertions:
    - result.code ShouldEqual 0
    - result.systemout ShouldContainSubstring bar

Builtin venom variables

name: MyTestSuite
testcases:
- name: testA
  steps:
  - type: exec
    script: echo '{{.venom.testsuite}} {{.venom.testsuite.filename}} {{.venom.testcase}} {{.venom.teststep.number}} {{.venom.datetime}} {{.venom.timestamp}}'
    # will display something as: MyTestSuite MyTestSuiteWithVenomBuiltinVar.yml testA 0 2018-08-05T21:38:24+02:00 1533497904

Builtin variables:

  • {{.venom.testsuite}}
  • {{.venom.testsuite.filename}}
  • {{.venom.testcase}}
  • {{.venom.teststep.number}}
  • {{.venom.datetime}}
  • {{.venom.timestamp}}

Venom templating

Beside venom variables, it is possible to use templating functions:

  • expandEnv : {{expandEnv }}, rewrites the named file and replaces ${var} or $var in the string according to the values of the current environment variables. References to undefined variables are replaced by the empty string. You can use it a script step for instance: script: cat {{expandEnv ./myFile}}.
Testsuite Versions
Version 2

On this new version, venom use the testsuite folder as the basepath instead of location of venom execution.

Considering this workspace:


tests/
   testsuiteA/
      testsuite.yml
      testa.json

On version 1


name: TestSuite Read File
testcases:
- name: TestCase Read File
  steps:
  - type: readfile
    path: testa.json
    assertions:
      - result.contentjson.foo ShouldEqual bar

If you execute venom run * into tests/ folder, venom will try to find testa.json on tests/testa.json and will failed. You must execute venom on the testsuite dir.

On version 2, venom use as basepath the testsuite file. So no matter where you execute venom command, testa.json will be found.

To specify the version 2, add version property on the testsuite:


version: "2"
name: TestSuite Read File
testcases:
- name: TestCase Read File
  steps:
  - type: readfile
    path: testa.json
    assertions:
      - result.contentjson.foo ShouldEqual bar

RUN Venom locally on CDS Integration Tests

cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var cdsro='cds -f $HOME/.cds/it.user.ro.json' --var cds='cds -f $HOME/.cds/it.user.rw.json' --parallel=5

RUN Venom with file var

vars.yaml :

cdsro: 'cds -f $HOME/.cds/it.user.ro.json'
cds: 'cds -f $HOME/.cds/it.user.rw.json'

cd $GOPATH/src/github.com/ovh/cds/tests
venom run --var-from-file vars.yaml --parallel=5

RUN Venom, with an export xUnit

venom run --format=xml --output-dir="."

Assertion

Keywords
  • ShouldEqual
  • ShouldNotEqual
  • ShouldAlmostEqual
  • ShouldNotAlmostEqual
  • ShouldResemble
  • ShouldNotResemble
  • ShouldPointTo
  • ShouldNotPointTo
  • ShouldBeNil
  • ShouldNotBeNil
  • ShouldBeTrue
  • ShouldBeFalse
  • ShouldBeZeroValue
  • ShouldBeGreaterThan
  • ShouldBeGreaterThanOrEqualTo
  • ShouldBeLessThan
  • ShouldBeLessThanOrEqualTo
  • ShouldBeBetween
  • ShouldNotBeBetween
  • ShouldBeBetweenOrEqual
  • ShouldNotBeBetweenOrEqual
  • ShouldContain
  • ShouldNotContain
  • ShouldContainKey
  • ShouldNotContainKey
  • ShouldBeIn
  • ShouldNotBeIn
  • ShouldBeEmpty
  • ShouldNotBeEmpty
  • ShouldHaveLength
  • ShouldStartWith
  • ShouldNotStartWith
  • ShouldEndWith
  • ShouldNotEndWith
  • ShouldBeBlank
  • ShouldNotBeBlank
  • ShouldContainSubstring
  • ShouldNotContainSubstring
  • ShouldEqualWithout
  • ShouldEqualTrimSpace
  • ShouldHappenBefore
  • ShouldHappenOnOrBefore
  • ShouldHappenAfter
  • ShouldHappenOnOrAfter
  • ShouldHappenBetween
  • ShouldHappenOnOrBetween
  • ShouldNotHappenOnOrBetween
  • ShouldHappenWithin
  • ShouldNotHappenWithin
  • ShouldBeChronological
  • ShouldNotExist

Most assertion keywords documentation can be found on smartystreets/assertions README.

Write your executor

An executor have to implement this interface


// Executor execute a testStep.
type Executor interface {
	// Run run a Test Step
	Run(ctx context.Content, venom.Logger, TestStep) (ExecutorResult, error)
}

Example



// Name of executor
const Name = "myexecutor"

// New returns a new Executor
func New() venom.Executor {
	return &Executor{}
}

// Executor struct
type Executor struct {
	Command string `json:"command,omitempty" yaml:"command,omitempty"`
}

// Result represents a step result
type Result struct {
	Code        int    `json:"code,omitempty" yaml:"code,omitempty"`
	Command     string `json:"command,omitempty" yaml:"command,omitempty"`
	Systemout   string   `json:"systemout,omitempty" yaml:"systemout,omitempty"` // put in testcase.Systemout by venom if present
	Systemerr   string   `json:"systemerr,omitempty" yaml:"systemerr,omitempty"` // put in testcase.Systemerr by venom if present
	Executor    Executor `json:"executor,omitempty" yaml:"executor,omitempty"`
}

// GetDefaultAssertions return default assertions for this executor
// Optional
func (Executor) GetDefaultAssertions() venom.StepAssertions {
	return venom.StepAssertions{Assertions: []string{"result.code ShouldEqual 0"}}
}

// Run execute TestStep
func (Executor) Run(ctx context.Context, l venom.Logger, step venom.TestStep) (venom.ExecutorResult, error) {

	// transform step to Executor Instance
	var e Executor
	if err := mapstructure.Decode(step, &e); err != nil {
		return nil, err
	}

	// to something with e.Command here...
	//...

	systemout := "foo"
	ouputCode := 0

	// prepare result
	r := Result{
		Code:    ouputCode, // return Output Code
		Command: e.Command, // return Command executed
		Systemout: systemout, // return Output string
		Executor: e, // return executor, useful for display Executor context in failure
	}

	return dump.ToMap(r)
}

Feel free to open a Pull Request with your executors.

TestCase Context

TestCase Context allows you to inject data in all steps.

Define a context is optional, but can be useful to keep data between test steps on a testcase.

Write your TestCase Context

A TestCase Context has to implement this interface


type TestCaseContext interface {
	Init() error
	Close() error
	SetTestCase(tc TestCase)
	GetName() string
}

Example

// Context Type name
const Name = "default"

// New returns a new TestCaseContext
func New() venom.TestCaseContext {
	ctx := &DefaultTestCaseContext{}
	ctx.Name = Name
	return ctx
}

// DefaultTestCaseContext represents the context of a testcase
type DefaultTestCaseContext struct {
	venom.CommonTestCaseContext
	datas map[string]interface{}
}

// Init Initialize the context
func (tcc *DefaultTestCaseContext) Init() error {
	return nil
}

// Close the context
func (tcc *DefaultTestCaseContext) Close() error {
	return nil
}

Methods SetTestCase and GetName are implemented by CommonTestCaseContext

Dependencies

Individual packages were updated using the rough procedure:

  1. dep ensure
  2. dep ensure -update ${PACKAGE}
  3. dep prune
  4. go build

Hacking

You've developed a new cool feature? Fixed an annoying bug? We'd be happy to hear from you! Make sure to read CONTRIBUTING.md before.

License

This work is under the BSD license, see the LICENSE file for details.

Documentation

Index

Constants

View Source 
const (
	// DetailsLow prints only summary results
	DetailsLow = "low"
	// DetailsMedium summary with lines in failure
	DetailsMedium = "medium"
	// DetailsHigh all
	DetailsHigh = "high"
)

Variables

View Source 
var (
	//Version is set with -ldflags "-X github.com/ovh/venom/venom.Version=$(VERSION)"
	Version = "snapshot"
)

Functions

func RemoveNotPrintableChar added in v0.16.0 

func RemoveNotPrintableChar(in string) string

RemoveNotPrintableChar removes not printable chararacter from a string

func ShouldContainSubstring added in v0.0.4 

func ShouldContainSubstring(actual interface{}, expected ...interface{}) string

ShouldContainSubstring receives exactly more than 2 string parameters and ensures that the first contains the second as a substring.

Types

type Aliases 

type Aliases map[string]string

Aliases contains list of aliases

type AssignStep added in v0.27.0 

type AssignStep struct {
	Assignments map[string]Assignment `json:"vars" yaml:"vars" mapstructure:"vars"`
}

type Assignment added in v0.27.0 

type Assignment struct {
	From  string `json:"from" yaml:"from"`
	Regex string `json:"regex" yaml:"regex"`
}

type CommonTestCaseContext 

type CommonTestCaseContext struct {
	TestCaseContext
	TestCase TestCase
	Name     string
}

CommonTestCaseContext represents a Default TestCase Context

func (*CommonTestCaseContext) GetName 

func (tcc *CommonTestCaseContext) GetName() string

GetName Get the context name

func (*CommonTestCaseContext) SetTestCase 

func (tcc *CommonTestCaseContext) SetTestCase(tc TestCase)

SetTestCase set testcase in context

type Executor 

type Executor interface {
	// Run run a Test Step
	Run(TestCaseContext, Logger, TestStep, string) (ExecutorResult, error)
}

Executor execute a testStep.

type ExecutorResult 

type ExecutorResult map[string]interface{}

ExecutorResult represents an executor result on a test step

type ExecutorWrap added in v0.0.7 

type ExecutorWrap struct {
	// contains filtered or unexported fields
}

ExecutorWrap contains an executor implementation and some attributes

type Failure 

type Failure struct {
	Value   string         `xml:",cdata" json:"value" yaml:"value,omitempty"`
	Result  ExecutorResult `xml:"-" json:"-" yaml:"-"`
	Type    string         `xml:"type,attr,omitempty" json:"type" yaml:"type,omitempty"`
	Message string         `xml:"message,attr,omitempty" json:"message" yaml:"message,omitempty"`
}

Failure contains data related to a failed test.

type H added in v0.27.0 

type H map[string]string

func ProcessVariableAssigments added in v0.27.0 

func ProcessVariableAssigments(tcName string, tcVars H, stepIn TestStep, l Logger) (H, bool, error)

func (*H) Add added in v0.27.0 

func (h *H) Add(k, v string)

func (*H) AddAll added in v0.27.0 

func (h *H) AddAll(h2 H)

func (*H) AddAllWithPrefix added in v0.27.0 

func (h *H) AddAllWithPrefix(p string, h2 H)

func (*H) AddWithPrefix added in v0.27.0 

func (h *H) AddWithPrefix(p, k, v string)

func (H) Clone added in v0.27.0 

func (h H) Clone() H

func (H) Get added in v0.27.0 

func (h H) Get(k string) string

type InnerResult 

type InnerResult struct {
	Value string `xml:",cdata" json:"value" yaml:"value"`
}

InnerResult is used by TestCase

type Logger added in v0.0.7 

type Logger interface {
	Debugf(format string, args ...interface{})
	Infof(format string, args ...interface{})
	Warnf(format string, args ...interface{})
	Warningf(format string, args ...interface{})
	Errorf(format string, args ...interface{})
	Fatalf(format string, args ...interface{})
}

Logger is basically an interface for logrus.Entry

type Property 

type Property struct {
	XMLName xml.Name `xml:"property" json:"-" yaml:"-"`
	Name    string   `xml:"name,attr" json:"name" yaml:"-"`
	Value   string   `xml:"value,attr" json:"value" yaml:"-"`
}

Property represents a key/value pair used to define properties.

type Skipped added in v0.16.0 

type Skipped struct {
	Value string `xml:",cdata" json:"value" yaml:"value,omitempty"`
}

Skipped contains data related to a skipped test.

type StepAssertions 

type StepAssertions struct {
	Assertions []string `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}

StepAssertions contains step assertions

type StepExtracts 

type StepExtracts struct {
	Extracts map[string]string `json:"extracts,omitempty" yaml:"extracts,omitempty"`
}

StepExtracts contains "step extracts"

type Templater 

type Templater struct {
	Values map[string]string
}

Templater contains templating values on a testsuite

func (*Templater) Add 

func (tmpl *Templater) Add(prefix string, values map[string]string)

Add add data to templater

func (*Templater) ApplyOnMap added in v0.21.1 

func (tmpl *Templater) ApplyOnMap(mapStringInterface map[string]interface{}) (bool, map[string]interface{}, error)

ApplyOnMap executes the template on a context return true if there is an variable replaced

func (*Templater) ApplyOnStep added in v0.0.7 

func (tmpl *Templater) ApplyOnStep(stepNumber int, step TestStep) (TestStep, error)

ApplyOnStep executes the template on a test step

type TestCase 

type TestCase struct {
	XMLName   xml.Name               `xml:"testcase" json:"-" yaml:"-"`
	Classname string                 `xml:"classname,attr,omitempty" json:"classname" yaml:"-"`
	Errors    []Failure              `xml:"error,omitempty" json:"errors" yaml:"errors,omitempty"`
	Failures  []Failure              `xml:"failure,omitempty" json:"failures" yaml:"failures,omitempty"`
	Name      string                 `xml:"name,attr" json:"name" yaml:"name"`
	Skipped   []Skipped              `xml:"skipped,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Status    string                 `xml:"status,attr,omitempty" json:"status" yaml:"status,omitempty"`
	Systemout InnerResult            `xml:"system-out,omitempty" json:"systemout" yaml:"systemout,omitempty"`
	Systemerr InnerResult            `xml:"system-err,omitempty" json:"systemerr" yaml:"systemerr,omitempty"`
	Time      string                 `xml:"time,attr,omitempty" json:"time" yaml:"time,omitempty"`
	TestSteps []TestStep             `xml:"-" hcl:"step" json:"steps" yaml:"steps"`
	Context   map[string]interface{} `xml:"-" json:"-" yaml:"context,omitempty"`
}

TestCase is a single test case with its result.

type TestCaseContext 

type TestCaseContext interface {
	Init() error
	Close() error
	SetTestCase(tc TestCase)
	GetName() string
}

TestCaseContext represents the context of a testcase

type TestStep 

type TestStep map[string]interface{}

TestStep represents a testStep

type TestSuite 

type TestSuite struct {
	XMLName    xml.Name               `xml:"testsuite" json:"-" yaml:"-"`
	Disabled   int                    `xml:"disabled,attr,omitempty" json:"disabled" yaml:"-"`
	Errors     int                    `xml:"errors,attr,omitempty" json:"errors" yaml:"-"`
	Failures   int                    `xml:"failures,attr,omitempty" json:"failures" yaml:"-"`
	Hostname   string                 `xml:"hostname,attr,omitempty" json:"hostname" yaml:"-"`
	ID         string                 `xml:"id,attr,omitempty" json:"id" yaml:"-"`
	Name       string                 `xml:"name,attr" json:"name" yaml:"name"`
	Filename   string                 `xml:"-" json:"-" yaml:"-"`
	ShortName  string                 `xml:"-" json:"-" yaml:"-"`
	Package    string                 `xml:"package,attr,omitempty" json:"package" yaml:"-"`
	Properties []Property             `xml:"-" json:"properties" yaml:"-"`
	Skipped    int                    `xml:"skipped,attr,omitempty" json:"skipped" yaml:"skipped,omitempty"`
	Total      int                    `xml:"tests,attr" json:"total" yaml:"total,omitempty"`
	TestCases  []TestCase             `xml:"testcase" hcl:"testcase" json:"tests" yaml:"testcases"`
	Version    string                 `xml:"version,omitempty" hcl:"version" json:"version" yaml:"version,omitempty"`
	Time       string                 `xml:"time,attr,omitempty" json:"time" yaml:"-"`
	Timestamp  string                 `xml:"timestamp,attr,omitempty" json:"timestamp" yaml:"-"`
	Vars       map[string]interface{} `xml:"-" json:"-" yaml:"vars"`
	Templater  *Templater             `xml:"-" json:"-" yaml:"-"`
	WorkDir    string                 `xml:"-" json:"-" yaml:"-"`
}

TestSuite is a single JUnit test suite which may contain many testcases.

type Tests 

type Tests struct {
	XMLName      xml.Name    `xml:"testsuites" json:"-" yaml:"-"`
	Total        int         `xml:"-" json:"total"`
	TotalOK      int         `xml:"-" json:"ok"`
	TotalKO      int         `xml:"-" json:"ko"`
	TotalSkipped int         `xml:"-" json:"skipped"`
	TestSuites   []TestSuite `xml:"testsuite" json:"test_suites"`
}

Tests contains all informations about tests in a pipeline build

type Venom added in v0.17.0 

type Venom struct {
	LogLevel  string
	LogOutput io.Writer

	PrintFunc func(format string, a ...interface{}) (n int, err error)

	IgnoreVariables []string
	Parallel        int

	EnableProfiling bool
	OutputFormat    string
	OutputDir       string
	StopOnFailure   bool
	// contains filtered or unexported fields
}

func New added in v0.17.0 

func New() *Venom

func (*Venom) AddVariables added in v0.17.0 

func (v *Venom) AddVariables(variables map[string]string)

func (*Venom) ContextWrap added in v0.17.0 

func (v *Venom) ContextWrap(tc *TestCase) (TestCaseContext, error)

ContextWrap initializes a context for a testcase no type -> parent context

func (*Venom) OutputResult added in v0.17.0 

func (v *Venom) OutputResult(tests Tests, elapsed time.Duration) error

OutputResult output result to sdtout, files...

func (*Venom) Parse added in v0.17.0 

func (v *Venom) Parse(path []string, exclude []string) error

Parse parses tests suite to check context and variables

func (*Venom) Process added in v0.17.0 

func (v *Venom) Process(path []string, exclude []string) (*Tests, error)

Process runs tests suite and return a Tests result

func (*Venom) RegisterExecutor added in v0.17.0 

func (v *Venom) RegisterExecutor(name string, e Executor)

RegisterExecutor register Test Executors

func (*Venom) RegisterTestCaseContext added in v0.17.0 

func (v *Venom) RegisterTestCaseContext(name string, tcc TestCaseContext)

RegisterTestCaseContext new register TestCaseContext

func (*Venom) RunTestStep added in v0.17.0 

func (v *Venom) RunTestStep(tcc TestCaseContext, e *ExecutorWrap, ts *TestSuite, tc *TestCase, stepNumber int, step TestStep, l Logger) ExecutorResult

RunTestStep executes a venom testcase is a venom context

func (*Venom) WrapExecutor added in v0.17.0 

func (v *Venom) WrapExecutor(t map[string]interface{}, tcc TestCaseContext) (*ExecutorWrap, error)

WrapExecutor initializes a test by name no type -> exec is default

Source Files

View all Source files

Directories

Path Synopsis
cli
venom
venom/run
venom/template
venom/update
venom/version
context
default
redis
webctx
dbfixtures
exec
grpc
http
imap
kafka
ovhapi
rabbitmq
readfile
redis
smtp
sql
ssh
web

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.