Automated Acceptance Tests
ui subdirectory contains UI functional tests which can be executed in a completely automated fashion using Capybara in a Docker container.
api subdirectory contains a set of tests for v2 of the CC REST API.
The tests may be run against Firefox or Chrome. It also includes support for screenshots of failed tests cases.
#Table of Contents
- How to run
- Known Issues
The docker image
zenoss/capybara contains all of the tools and libraries required to run Capybara against Firefox or Chrome.
ui is mounted into the docker container under the directory
/capybara, giving the tools in the container access to all of the cucumber/capybara test files defined in
runCucumber.sh is executed from within docker. It handles any runtime setup (such as starting Xvfb if necessary), and
then executes Cucumber.
A report of the test execution is written to
How to run
Step 1 - Start Control Center
The test suite assumes serviced is already running and ready to receive requests from a web browser or a REST client.
To run the test, you must specify the URL for Control Center either
by setting the environment variable APPLICATION_URL or by specifying the
command line argument
-a <url>. Do not use https://localhost because from
the perspective of Capybara/Cucumber running inside the container, https://localhost
refers to the container itself.
Step 2 - Build the docker image (optional)
The docker image containing Cucumber and Capybara is available on the Zenoss dockerhub repo as
zenoss/capybara:$(VERSION), so you do not need to build it yourself.
In case you do want/need to build the docker image containing Cucumber and Capybara, use the following commands:
$ zendev cd serviced $ cd acceptance/dockerImage $ make
Once the image has been built, you can use the following commands to push it to Docker hub:
$ zendev cd serviced $ cd acceptance/dockerImage $ make dockerPush
NOTE: The version of the docker image is defined by the file dockerImage/VERSION. If you modify the contents of the image, you should update the version number in that file before building/pushing a new image.
Step 3 - Setup the test environment
Start the mock agents
The test suite uses mock agents for tests involving hosts. The mock agents do not (currently) connect to Zookeeper, nor do they have the ability to mock service executions. Their primary purpose is to provide light-weight mocks for the set of properties for each host (CPU, RAM, Kernel Version, CC version, etc). To build the mock agent, use the following commands:
$ zendev cd serviced $ make mockAgent
To run the mock agents, use the following commands:
$ cd acceptance $ ./startMockAgents.sh
This step is not necessary if you do not run tests involving hosts.
NOTE: If you stop and restart the start mock agents script while serviced is running, you may see a "Bad Request: connection is shut down" error when you try to add a mock agent. Restarting serviced will fix this.
Add the test template
The Application tests assume that a test template has already been added to the system. To compile and add the test template, use the following commands:
$ zendev cd serviced $ serviced template compile dao/testsvc | serviced template add
In the future, this step may be incorporated directly into the Cucumber tests for Applications.
Step 4 - Run the test suite
Note: on a local developer setup, please set the following in your env or at cmd line as you run the launcher scripts:
SERVICED_ETC_PATH=$(zendev root)/opt_serviced/etc SERVICED_ISVCS_PATH=$(zendev root)/opt_serviced/var/isvcs SERVICED_BINARY=$(which serviced)
Capybara uses different 'drivers' to interface with a web browser.
runUIAcceptance.sh executes tests against Chrome.
The test suite can be run against any one of several browsers by selecting different drivers.
Both of the following commands run the test suite against Chrome:
$ ./runUIAcceptance.sh -a <servicedURL> -u <userID> -p <password> or $ ./runUIAcceptance.sh -d selenium_chrome -a <servicedURL> -u <userID> -p <password>
To run the tests against Firefox, use
$ ./runUIAcceptance.sh -d selenium -a <servicedURL> -u <userID> -p <password>
For a full description of the command line options, run
None of the above browser concerns are needed in the REST tests.
$ ./runAPIAcceptance.sh -a <servicedURL> -u <userID> -p <password>
Step 5 - Review the test results
The output from the tests are written to stdout as the tests execute. Additionally, an HTML report is written to
ui/output/feature-overview.html. If a test case fails, the HTML report will include a screenshot of the browser
at the point in time when the test case failed.
Cucumber Command Line Options
Cucumber command line options can be specified by defining the environment variable
CUCUMBER_OPTS on the make command line.
Environment variables are used to pass information into the docker container which are used by the shell and Ruby scripts executed in the container.
The primary variables used by
APPLICATION_URL- the URL of the application under test. You can set this variable with the
-acommand line option for
APPLICATION_USERID- the user id to login into the application under test. You can set this variable with the
-ucommand line option for
APPLICATION_PASSWORD- the password used to login into the application under test. You can set this variable with the
-pcommand line option for
CAPYBARA_DRIVER- the name of the Capybara web driver to use. Valid values are
selenium(which uses Firefox), or
selenium_chrome. The default if not specified is
selenium_chrome. You can set this variable with the
-dcommand line option for
CAPYBARA_TIMEOUT- the timeout, in seconds, that Capybara should wait for a page or element. The default is 10 seconds. You can set this variable with the
-tcommand line option for
CUCUMBER_OPTS- any of the standard command line options for Cucumber.
DATASET- the JSON dataset to use as test input. You can set this variable with the
--datasetcommand line option for
Internally, the script also uses the variables
CALLER_GID to capture the current user's UID and GID which are used in the container so that
files written to
ui/output will have the proper owner/group information (for OSX, see Known Issues). These two variables should not be overwritten or modified. For an example of how these two variables are used, refer to the dockerImage/build/runCucumber.sh script.
For a full list of possible options for Cucumber itself, pass the
--help option to Cucumber like this:
$ CUCUMBER_OPTS=--help ./runUIAcceptance.sh -a <servicedURL> -u <userid> -p <password>
Running a subset of tests
Cucumber supports a feature called tags which can be used to run a subset of tests. A full explanation of using tags is beyond the scope of this document; see the Cucumber documentation for a full description.
For example, you can run tests for a single tag with a command like:
$ ./runUIAcceptance.sh -a <servicedURL> -u <userid> -p <password> --tags @hosts
or you can include tags as part of the
CUCUMBER_OPTS environment variable
$ CUCUMBER_OPTS='--tags @hosts' ./runUIAcceptance.sh -a <servicedURL> -u <userid> -p <password>
Looking at a failed test case
If a test step fails, the test harness will capture a screenshot. If you have not seen a failed test case already, run the tests with an invalid userid and/or password. If you open
ui/output/feature-overview.html with a browser and drill into the details for the login feature, you will see the output in a nicely formatted table. Look for a "Screenshot" link below the error report for the failed step. Clicking on that link should display an image captured at the time of the failure.
Cucumber offers a powerful tagging feature that can be used to control which features and/or scenarios are run, as well as enabling custom 'hooks' to run specific blocks of code at different points.
Some of the tags defined by this project are:
- feature tags - There is a unique tag on each Feature to allow running single features at a time or in combination. Some of the valid values are
- login hook - The tag
@login-requiredillustrates how to use hook tags to automatically execute some block of code before/after the associated feature/step. In this case, the
@login-requiredtag will login the user before each feature/step decorated with the tag (remember that each Scenario executes as a new browser session).
To specify one or more of these tags, add the
--tags tagName to the command line. For instance, the following command will run just the tests for the hosts feature:
$ ./runUIAcceptance.sh -a servicedURL -u yourUserID -p yourPasswordHere --tags @hosts
For information about this Cucumber feature, see:
Watching the browser while the tests run
Normally, the browser used by the tests is executed within the Docker container using
the X virtual framebuffer (Xvfb) so nothing is displayed. For debugging purposes,
it is often useful to view the browser while the test runs. To do that, include the
argument as illustrated below:
$ xhost + $ ./runUIAcceptance.sh --debug -a servicedURL -u yourUserID -p yourPasswordHere
NOTE: You only need to execute the command
xhost + once.
Mark a test PENDING
When a test is marked
PENDING, Cucumber will skip it. This is useful for several scenarios, including
Behavior Driven Development (BDD), where you want to first write the test case, and then write the
code to make the test case pass. However, in between
the time you create the test case and the time you implement the code, you may not want all of
the intervening test-case executions to fail (e.g. tests launched by an automated build).
The following example illustrates how to mark a test case as PENDING:
@login-required Scenario: View empty Hosts page When PENDING I am on the hosts page Then I should see "Applications" And I should see "No Data Found" And I should see "0 Results"
Page Object Model
The combination of Cucumber and Capybara offer huge advantages in terms of being able to write tests using a simple, expressive DSL that describes how to interact with your application's UI. However, as your application and your tests grow, you can easily run into situations where implementation details about a particular page or reusable element are either 'leaking' into Step statements explicitly, or they are constantly repeated across step definitions. For instances, things like the IDs or CSS/Xpath expressions used to identify specific elements on a page can end up being repeated over and over again. When the actual page definition is changed by a developer, then you have to make the same refactor across multiple places in your tests.
A Page Object Model is a DSL for describing for the page itself. Think of it as a secondary DSL "below" the DSL expresssed in the test features. The Page Object Model should encapsulate all of the implementation detail like DOM identifiers, CSS/xpath matching expressions, etc.
These tests use Site Prism to implement a Page Object Model. The various page objects are is defined in the ui/features/pages directory and used in the step definitions. For instance, the login page is defined in ui/features/pages/login.rb, and it is used in ui/features/steps/login_steps.rb.
For more discussion of the Page Object Model, see
Please follow these guidelines when writing or modifying tests.
- Add example of CLI validation using Cucumber.
- The code in 'common' is an altered copy of the CLI wrapper specific to the UI tests. There were gem dependency issues (mime-types) when trying to reuse the code as is -- maybe fork cucumber-api and try updating the rest-client gem.
- In order to share common code, the 'common' folder is mounted and symlinked in the container. There is a file called common_include that is pulling in dependencies from common, since cucumber can't find them even though they are under the 'features' folder.'
- With upgrade to zenoss/capybara:1.1.0, the --debug option stopped working.
- The tests don't work on Mac OSX for a variety of reasons:
- The run script makes Linux-specific assumptions about mapping timezone definitions into the container.
- On Mac OSX with boot2docker, if you have problems reaching archive.ubuntu.com while trying to run
dockerBuild.sh, refer to the workaround here.
- On Mac OSX with boot2docker, boot2docker will automagically map the root user of the docker container to the current user of the host OS. This use-case needs more testing.
- Cucumber - A tool for BDD testing
- The Cucumber Book by Aslak Hellesoy, Matt Wynne
- Capybara - An Acceptance test framework for web applications
- Documentation for Capybara
- Application Testing with Capybara by Matthew Robbins
- Capybara cheat sheet
- Site Prism - A Page Object Model DSL for Capybara
- Notes on setting up Xvfb for docker
- How to install Chromedriver on Ubuntu
- How to install Chrome from the command line
- URLs for different versions of Chrome
- Chromedriver options
- Chrome options