Documentation
¶
Overview ¶
Package suite contains functions to input tests from disk and execute suites of tests in a controlled way.
Variable Handling ¶
There are three "scopes" for variables when executing a suite read from disk (a RawSuite):
Global Scope: Variables set from "the outside", typically via the -D command line flag to cmd/ht. Executing a suite or tests does not modify variables in this scope. This global set of varibales is input to the Execute method of RawSuite.
Suite Scope: This is the context of the suite, it can be changed dynamically via variable extraction from executed tests. The initial suite scope is:
a copy of the Global Scope, with
added COUNTER and RANDOM variables and
merged defaults from the suite's Variables section. Variable expansion happens inside the values of the Variables section.
Test Scope: The tests scope is the context of a test, actually the set of variables used to make a ht.Test from a RawTest. It is constructed as follows:
To a copy of the actual suite scope,
the automatic variables COUNTER and RANDOM are added (with new values),
merged with the call Variables
merged with the tests Variables section. Variable expansion happens inside the values of the call and test Variables section.
Here "merging" means that the variables in the Variables section are added to the scope if not already present. I.e. the variables from outer scope dominate variables from inner scopes.
Throughput Load-Testing ¶
Load tests in the form of a throuhghput-test can be made through the Throughput function. This function will try to create a certain throughput load, i.e. a certain average number of request per second. The intervalls between request follow an exponential distribution which mimics the load generated from real-world, uncorrelated users.
The requests are generated from different Scenarios which contribute a certain percentage of requests to the set of all requests. The scenarios are basically just suites of tests: One suite might simulate the bahaviour of a bot while an other scenario can simulate the behaviour of a "normal" user and a third scenario performs actions a user with special interests.
The Tests of each suite/scenario are executed, including the checks. Note that some checks can produce additional requests which are not accounted in the throughput rate of the load test (but do hit the target server). Checks can be turned off on per scenario basis.
If all tests (and thus request) in a suite/scenario have been executed, the suite is repeated. To reach the desired request througput rate each scenario is run in multiple parallel threads. New threads are started as needed. The number of threads may be limited on a per scenario basis.
Index ¶
- Variables
- func DataPrint(data []bender.TestData, out io.Writer)
- func DataToCSV(data []bender.TestData, out io.Writer) error
- func HTMLReport(dir string, s *Suite) error
- type Executor
- type File
- type FileSystem
- type LoopIteration
- type Mixin
- type RawElement
- type RawLoadTest
- type RawScenario
- type RawSuite
- type RawTest
- type Scenario
- type Suite
Constants ¶
This section is empty.
Variables ¶
var ( SuiteTmpl *template.Template ShortSuiteTmpl *template.Template HtmlSuiteTmpl *htmltemplate.Template )
var (
ErrAbortExecution = errors.New("Abort Execution")
)
var GetCounter <-chan int
GetCounter returns a strictly increasing sequence of int values.
var IDSep = "\u2237"
IDSep is the separator string used in constructing IDs for the individual test executed. The test Name are (miss)used to report details:
<ScenarioNo>/<ThreadNo>/<Repetition>/<TestNo> IDSep <ScenarioName> IDSep <TestName>
Functions ¶
func HTMLReport ¶
HTMLReport generates a report of the outcome of s to directory dir.
Types ¶
type Executor ¶
A Executor is responsible for executing the given test during the Iterate'ion of a Suite. It should return nil if execution should continue and ErrAbortExecution to stop further iteration.
type File ¶
File is a textual representation of a hjson data read from disk.
type FileSystem ¶
FileSystem acts like an in-memory filesystem. A empty FileSystem accesses the real OS file system.
func NewFileSystem ¶
func NewFileSystem(txt string) (FileSystem, error)
NewFileSystem parses txt which must be of the form
# <filename1> <filecontent1> # <filename2> <filecontent2> ...
into a new FileSystem.
type LoopIteration ¶
type LoopIteration struct {
Data interface{}
I int // 0-based loop index
N int // 1-based loop index
Odd, Even bool
}
LoopIteration helps ranging over Data in a template.
type RawElement ¶
RawElement represents one test in a RawSuite.
type RawLoadTest ¶
type RawLoadTest struct {
*File
Name string
Description string
Scenarios []RawScenario
Variables map[string]string
}
RawLoadTest as read from disk.
func LoadRawLoadtest ¶
func LoadRawLoadtest(filename string, fs FileSystem) (*RawLoadTest, error)
LoadRawLoadtest from the given filename.
func (*RawLoadTest) ToScenario ¶
func (raw *RawLoadTest) ToScenario(globals map[string]string) []Scenario
ToScenario produces a list of scenarios from raw.
type RawScenario ¶
type RawScenario struct {
Name string // Name of this Scenario
File string // File is the RawSuite to use as scenario
Percentage int // Percantage this scenario contributes to the load test.
MaxThreads int // MaxThreads to use for this scenario. 0 means unlimited.
Variables map[string]string // Variables used.
OmitChecks bool // OmitChecks in the tests.
// contains filtered or unexported fields
}
RawScenario represents a scenario in a load test.
type RawSuite ¶
type RawSuite struct {
*File
Name, Description string
Setup, Main, Teardown []RawElement
KeepCookies bool
OmitChecks bool
Variables map[string]string
Verbosity int
// contains filtered or unexported fields
}
RawSuite represents a suite as represented on disk as a HJSON file.
func LoadRawSuite ¶
func LoadRawSuite(filename string, fs FileSystem) (*RawSuite, error)
LoadRawSuite with the given filename from fs.
func (*RawSuite) AddRawTests ¶
AddRawTest adds ts to the tests in rs.
func (*RawSuite) Execute ¶
func (rs *RawSuite) Execute(global map[string]string, jar *cookiejar.Jar, logger *log.Logger) *Suite
Execute the raw suite rs and capture the outcome in a Suite.
Tests are executed linearely, first the Setup, then the Main and finaly the Teardown test. Any Failure or error during setup will skip further setup and main test (but allteardown will be executed). The following table shows two runs with possible outcomes.
Test Type Run1 Run2 ------------------------------------------ Setup-1 Pass Pass Setup-2 Fail Pass Setup-3 Skip Pass Main-1 Skip Fail Main-2 Skip Pass Teardown-1 Pass Pass Teardown-2 Fail Error Teardown-3 Pass Pass
type RawTest ¶
type RawTest struct {
*File
Mixins []*Mixin // Mixins of this test.
Variables map[string]string // Variables are the defaults of the variables.
// contains filtered or unexported fields
}
RawTest is a raw for of a test as read from disk with its mixins and its variables.
func LoadRawTest ¶
func LoadRawTest(filename string, fs FileSystem) (*RawTest, error)
LoadRawTest reads filename and produces a new RawTest.
type Scenario ¶
type Scenario struct {
// Name of this scenario. May differ from the suite.
Name string
// RawSuite is the suite to execute repeatedly to generate load.
*RawSuite
// Percentage of requests in the throughput test taken from this
// scenario.
Percentage int
// MaxThreads limits the number of threads used to generate load from
// this scenario. The value 0 indicates unlimited number of threads.
MaxThreads int
// OmitChecks turns off checks in this scenario, i.e. only the request
// is made but no checks are performed on the response.
OmitChecks bool
// The logger to use for tests of this scenario.
Log *log.Logger
// contains filtered or unexported fields
}
Scenario describes a single scenario to be run as part of a throughput test.
type Suite ¶
type Suite struct {
Name string // Name of the Suite.
Description string // Description of what's going on here.
KeepCookies bool // KeepCookies in a cookie jar common to all Tests.
Status ht.Status // Status is the overall status of the whole suite.
Error error // Error encountered during execution of the suite.
Started time.Time // Start of the execution.
Duration time.Duration // Duration of the execution.
Tests []*ht.Test // The Tests to execute
Variables map[string]string // The initial variable assignemnt
FinalVariables map[string]string // The final set of variables.
Jar *cookiejar.Jar // The cookie jar used
Log *log.Logger // The logger used.
// contains filtered or unexported fields
}
A Suite is a collection of Tests which can be executed sequentily with the result captured.
func NewFromRaw ¶
func NewFromRaw(rs *RawSuite, global map[string]string, jar *cookiejar.Jar, logger *log.Logger) *Suite
NewFromRaw sets up a new Suite from rs, read to be Iterated.
func Throughput ¶
func Throughput(scenarios []Scenario, rate float64, duration, ramp time.Duration) ([]bender.TestData, *Suite, error)
Throughput runs a throughput load test with an average rate of request/seconds for the given duration. The request are generated from the given scenarios. It returns a summary of all request, all failed tests collected into a Suite and an error indicating if the throughput test reached the targeted rate and distribution.
func (*Suite) JUnit4XML ¶
JUnit4XML generates a JUnit 4 compatible XML result with each Check reported as an individual testcase. NotRun checks are reported as Skipped and Bogus checks are counted as Errored tests.
func (*Suite) PrintReport ¶
PrintReport outputs a textual report of s to w.
func (*Suite) PrintShortReport ¶
PrintShortReport outputs a short textual report of s to w.
Source Files