test-network-function

module
Version: v1.0.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 11, 2021 License: GPL-2.0

README

Test Network Function build Go Report Card

This repository contains a set of network function test cases and the framework to build more. It also generates reports (claim.json) on the result of a test run. The tests and framework are intended to test the interaction of Cloud-Native Network Functions (CNFs) with OpenShift Container Platform.

The suite is provided here in part so that CNF Developers can use the suite to test their CNFs readiness for certification. Please see "CNF Developers" below for more information.

Container

Running

A container that can be used to run the tests is available from quay.io

To pull the latest container and run the tests you use the following command. There are several required arguments:

  • -t gives the local directory that contains tnf config files set up for the test.
  • -o gives the local directory that the test results will be available in once the container exits.
  • Finally, list the specs to be run must be specified, space-separated.

Optional arguments are:

  • -i gives a name to a custom TNF container image. Supports local images, as well as images from external registries.
  • -k gives a path to one or more kubeconfig files to be used by the container to authenticate with the cluster. Paths must be separated by a colon.
  • -n gives the network mode of the container. Defaults to bridge. See the docker run --network parameter reference for more information on how to configure network settings.

If -k is not specified, autodiscovery is performed. The autodiscovery first looks for paths in the $KUBECONFIG environment variable on the host system, and if the variable is not set or is empty, the default configuration stored in $HOME/.kube/config is checked.

./run-container.sh -k ~/.kube/config -t ~/tnf/config -o ~/tnf/output diagnostic generic

Note: Tests must be specified after all other arguments!

Building

You can build an image locally by using the command below. Use the value of TNF_VERSION to set a branch, a tag, or a hash of a commit that will be installed into the image.

docker build -t test-network-function:v1.0.9 --build-arg TNF_VERSION=v1.0.9 .

To build an image that installs TNF from an unofficial source (e.g. a fork of the TNF repository), use the TNF_SRC_URL build argument to override the URL to a source repository.

docker build -t test-network-function:v1.0.9 \
  --build-arg TNF_VERSION=v1.0.9 \
  --build-arg TNF_SRC_URL=https://github.com/test-network-function/test-network-function .

To make run-container.sh use the newly built image, specify the custom TNF image using the -i parameter.

./run-container.sh -i test-network-function:v1.0.9 -t ~/tnf/config -o ~/tnf/output diagnostic generic

Dependencies

At a minimum, the following dependencies must be installed prior to running make install-tools.

Dependency Minimum Version
GoLang 1.14
golangci-lint 1.32.2
jq 1.6
OpenShift Client 4.4

Other binary dependencies required to run tests can be installed using the following command:

make install-tools

Finally the source dependencies can be installed with

make update-deps

Note: You must also make sure that $GOBIN (default $GOPATH/bin) is on your $PATH.

Note: Efforts to containerize this offering are considered a work in progress.

Available Test Specs

There are two categories for CNF tests; 'General' and 'CNF-specific'.

The 'General' tests are designed to test any commodity CNF running on OpenShift, and include specifications such as 'Default' network connectivity.

'CNF-specific' tests are designed to test some unique aspects of the CNF under test are behaving correctly. This could include specifications such as issuing a GET request to a web server, or passing traffic through an IPSEC tunnel.

General

The general-purpose category covers most tests. It consists of multiple suites that can be run in any combination as is appropriate for the CNF(s) under test:

Suite Test Spec Description Minimum OpenShift Version
diagnostic The diagnostic test suite is used to gather node information from an OpenShift cluster. The diagnostic test suite should be run whenever generating a claim.json file. 4.4.3
generic The generic test suite is used to test Default network connectivity between containers. It also checks that the base container image is based on RHEL. 4.4.3
multus The multus test suite is used to test SR-IOV network connectivity between containers. 4.4.3
operator The operator test suite is designed basic Kubernetes Operator functionality. 4.4.3
container The container test suite is designed to test container functionality and configuration 4.4.3

Further information about the current offering for each test spec is included below.

diagnostic tests

The diagnostic test spec issues commands to poll environment information which can be appended to the claim file. This information is necessary to ensure a properly spec'd environment is used, and allows the claim to be reproduced. As of today, the diagnostic test suite just polls Node information for all Nodes in the cluster. Future iterations may consider running lshw or similar types of diagnostic tests.

generic tests

The generic test spec tests:

  1. Default network connectivity between containers.
  2. That CNF container images are RHEL based.

To test Default network connectivity, a test partner pod is installed on the network. The test partner pod is instructed to issue ICMPv4 requests to each container listed in the test configuration, and vice versa. The test asserts that the test partner pod receives the correct number of replies, and vice versa.

In the future, other networking protocols aside from ICMPv4 should be tested.

multus tests

Similar to the generic test spec, the multus test spec is utilized for CNFs that utilize multiple network interfaces. As of today, the multus test suite just tests that a test partner pod can successfully ping the secondary interface of the CNF containers. Since SR-IOV is often utilized, and the secondary interface of a CNF cannot be accessed in user space, the test is unidirectional.

operator tests

Currently, the operator test spec is limited to a single test case called OPERATOR_STATUS. OPERATOR_STATUS just checks that the CSV corresponding to the CNF Operator is properly installed.

In the future, tests surrounding Operational Lifecycle Management will be added.

container tests

The container test spec has the following test cases:

Test Name Description
HOST_NETWORK_CHECK Ensures that the CNF pods do not utilize host networking. Note: This test can be disabled for infrastructure CNFs that should utilize host networking.
HOST_PORT_CHECK Ensures that the CNF pods do not utilize host ports.
HOST_PATH_CHECK Ensures that the CNF pods do not utilize the host filesystem.
HOST_IPC_CHECK Ensures that the CNF pods do not utilize host IPC namespace to access or control host processes.
HOST_PID_CHECK Ensures that the CNF pods do not utilize host PID namespace to access or control host processes.
CAPABILITY_CHECK Ensures that the CNF SCC is not configured to allow NET_ADMIN or SYS_ADMIN.
ROOT_CHECK Ensure that the CNF pods are not run as root.
PRIVILEGE_ESCALATION Ensure that the CNF SCC is not configured to allow privileged escalation.

In the future, we are considering additional tests to ensure aspects such as un-alteration of the container image.

Performing Tests

Currently, all available tests are part of the "CNF Certification Test Suite" test suite, which serves as the entrypoint to run all test specs. CNF Certification 1.0 is not containerized, and involves pulling, building, then running the tests.

By default, test-network-function emits results to test-network-function/cnf-certification-tests_junit.xml.

The included default configuration is for running generic and multus suites on the trivial example at cnf-certification-test-partner. To configure for your own environment, please see the Test Configuration section, below.

Pulling The Code

In order to pull the code, issue the following command:

mkdir ~/workspace
cd ~/workspace
git clone git@github.com:test-network-function/test-network-function.git
cd test-network-function
Building the Tests

In order to build the test executable, first make sure you have satisfied the dependencies.

make build-cnf-tests

Gotcha: The make build* commands run unit tests where appropriate. They do NOT test the CNF.

Testing a CNF

Once the executable is built, a CNF can be tested by specifying which suites to run using the run-cnf-suites.sh helper script. Any combintation of the suites listed above can be run, e.g.

./run-cnf-suites.sh diagnostic
./run-cnf-suites.sh diagnostic generic
./run-cnf-suites.sh diagnostic generic multus
./run-cnf-suites.sh diagnostic operator
./run-cnf-suites.sh diagnostic generic multus container operator

By default the claim file will be output into the same location as the test executable. The -o argument for run-cnf-suites.sh can be used to provide a new location that the output files will be saved to. For more detailed control over the outputs, see the output of test-network-function.test --help.

cd test-network-function && ./test-network-function.test --help

Gotcha: The generic test suite requires that the CNF has both ping and ip binaries installed. Please add them manually if the CNF under test does not include these. Automated installation of missing dependencies is targetted for a future version.

Test Configuration

There are a few pieces of configuration required to allow the test framework to access and test the CNF:

Config File Purpose
generic_test_configuration.yml Describes the CNF or CNFs that are to be tested, the container that will run the tests, and the test orchestrator.
cnf_test_configuration.yml Defines which containers and operators are to be tested, and in which roles.
testconfigure.yml Defines roles, and which tests are appropriate for which roles. It should not be necessary to modify this.

Combining these configuration files is a near-term goal.

generic_test_configuration.yml

The config file generic_test_configuration.yml contains three sections:

  • containersUnderTest: describes the CNFs that will be tested. Each container is defined by the combination of its namespace, podName, and containerName, which are also used to connect to the container when required.

    • Each entry for containersUnderTest must also define the defaultNetworkDevice of that container. There is also an optional multusIpAddresses that can be omitted if the multus tests are not run.
  • partnerContainers: describes the containers that support the testing. Multiple partnerContainers allows for more complex testing scenarios. At the time of writing, only one is used, which will also be the test orchestrator.

  • testOrchestrator: references a partner containers that is used for the generic test suite. The test partner is used to send various types of traffic to each container under test. For example the orchestrator is used to ping a container under test, and to be the ping target of a container under test.

The included example defines a single container to be tested, and a single partner to do the testing.

cnf_test_configuration.yml - Operator and Container Test Configuration

Testing operator and containers in specific roles is currently configured separately from the generic tests. This is configured using cnf_test_configuration.yml.

cnf_test_configuration.yml defines the roles under which operators and containers are to be tested.

The included example config is set up with some examples of this: It will run the "OPERATOR_STATUS" tests (as defined in testconfigure.yml) against an etcd operator, and the "PRIVILEGED_POD" and "PRIVILEGED_ROLE" tests against an nginx container.

A more extensive example is provided in pkg/config/example.yaml

Test Output

Claim File

The test suite generates a "claim" file, which describes the system(s) under test, the tests that were run, and the outcome of all of the tests. This claim file is the proof of the test run that is evaluated by Red Hat when "certified" status is being considered. For more information about the contents of the claim file please see the schema. You can read more about the purpose of the claim file and CNF Certification in the Guide.

Adding Test Results for the CNF Validation Test Suite to a Claim File

e.g. Adding a cnf platform test results to your existing claim file.

You can use the claim cli tool to append other related test suite results to your existing claim.json file. The output of the tool will be an updated claim file.

go run cmd/tools/cmd/main.go claim-add --claimfile=claim.json --reportdir=/home/$USER/reports

Args:
--claimfile is an existing claim.json file --repordir :path to test results that you want to include.

The tests result files from the given report dir will be appended under the result section of the claim file using file name as the key/value pair. The tool will ignore the test result, if the key name is already present under result section of the claim file.

 "results": {
 "cnf-certification-tests_junit": {
 "testsuite": {
 "-errors": "0",
 "-failures": "2",
 "-name": "CNF Certification Test Suite",
 "-tests": "14",
Command Line Output

When run the CNF test suite will output a report to the terminal that is primarily useful for Developers to evaluate and address problems. This output is similar to many testing tools.

Here's an example of a Test pass. It shows the Test running a command to extract the contents of /etc/redhat-release and using a regular expression to match allowed strings. It also prints out the string that matched.:

------------------------------
generic when test(test) is checked for Red Hat version 
  Should report a proper Red Hat version
  /Users/$USER/cnf-cert/test-network-function/test-network-function/generic/suite.go:149
2020/12/15 15:27:49 Sent: "if [ -e /etc/redhat-release ]; then cat /etc/redhat-release; else echo \"Unknown Base Image\"; fi\n"
2020/12/15 15:27:49 Match for RE: "(?m)Red Hat Enterprise Linux Server release (\\d+\\.\\d+) \\(\\w+\\)" found: ["Red Hat Enterprise Linux Server release 7.9 (Maipo)" "7.9"] Buffer: "Red Hat Enterprise Linux Server release 7.9 (Maipo)\n"
•

The following is the output from a Test failure. In this case, the test is checking that a CSV (ClusterServiceVersion) is installed correctly, but does not find it (the operator was not present on the cluster under test):

------------------------------
operator Runs test on operators when under test is: my-etcd/etcdoperator.v0.9.4  
  tests for: CSV_INSTALLED
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122
2020/12/15 15:28:19 Sent: "oc get csv etcdoperator.v0.9.4 -n my-etcd -o json | jq -r '.status.phase'\n"

• Failure [10.002 seconds]
operator
/Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:58
  Runs test on operators
  /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:71
    when under test is: my-etcd/etcdoperator.v0.9.4 
    /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:121
      tests for: CSV_INSTALLED [It]
      /Users/$USER/cnf-cert/test-network-function/test-network-function/operator/suite.go:122

      Expected
          <int>: 0
      to equal
          <int>: 1

CNF Developers

Developers of CNFs, particularly those targeting CNF Certification with Red Hat on OpenShift, can use this suite to test the interaction of their CNF with OpenShift. If you are interested in CNF Certification please contact Red Hat.

Refer to the rest of the documentation in this file to see how to install and run the tests as well as how to interpret the results.

You will need an OpenShift 4.4 installation running your CNF, and at least one other machine available to host the test suite. The cnf-certification-test-partner repository has a very simple example of this you can model your setup on.

Known Issues

Issue-161 Some containers under test do nto contain ping or ip binary utilities

In some cases, containers do not provide ping or ip binary utilities. Since these binaries are required for the connectivity tests, we must exclude such containers from the connectivity test suite. In order to exclude these containers, please issue add the following to test-network-function/generic_test_configuration.yaml:

excludeContainersFromConnectivityTests:
  - namespace: <namespace>
    podName: <podName>
    containerName: <containerName>

Note: Future work may involve installing missing binary dependencies.

Directories

Path Synopsis
cmd
catalog
Package main provides a CLI driven tool to generate the test catalog.
Package main provides a CLI driven tool to generate the test catalog.
catalog/cmd
Package cmd provides a CLI implementation for generating the test catalog in JSON or markdown.
Package cmd provides a CLI implementation for generating the test catalog in JSON or markdown.
claim
Package main provides a CLI driven tool to append test suite result to existing claim file.
Package main provides a CLI driven tool to append test suite result to existing claim file.
generic
Package main introduces an executable (jsontest) used to run JSON based tests.
Package main introduces an executable (jsontest) used to run JSON based tests.
oc
Package main provides an example of spawning an OpenShift client and using it to ping a target IP address.
Package main provides an example of spawning an OpenShift client and using it to ping a target IP address.
ping
Package main provides an example of pinging a target IP address.
Package main provides an example of pinging a target IP address.
ssh
Package main provides an example of establishing an SSH session
Package main provides an example of establishing an SSH session
pkg
config
Package config provides test-network-function configuration along with a config pool for aggregating configuration.
Package config provides test-network-function configuration along with a config pool for aggregating configuration.
junit
Package junit provides utility conversion methods for encoding JUnit XML results to generic JSON.
Package junit provides utility conversion methods for encoding JUnit XML results to generic JSON.
tnf
Package tnf contains the core Test runner interfaces.
Package tnf contains the core Test runner interfaces.
tnf/config
Package config provides configurations handler for cnf test framework.
Package config provides configurations handler for cnf test framework.
tnf/dependencies
Package dependencies provides constants related to test runtime dependencies.
Package dependencies provides constants related to test runtime dependencies.
tnf/handlers
Package handlers provides a variety of Test Handler implementations.
Package handlers provides a variety of Test Handler implementations.
tnf/handlers/base/redhat
Package redhat provides a tnf.Test implementation which tests whether a container is based on Red Hat technologies.
Package redhat provides a tnf.Test implementation which tests whether a container is based on Red Hat technologies.
tnf/handlers/container
Package container provides functionality to test pod and container level checks.
Package container provides functionality to test pod and container level checks.
tnf/handlers/generic
Package generic provides a simplified interface for creating a tnf.Test and corresponding reel.Handler from generic JSON input.
Package generic provides a simplified interface for creating a tnf.Test and corresponding reel.Handler from generic JSON input.
tnf/handlers/generic/assertion
Package assertion defines the ability for users to include assertions on the regular expression matched in ReelMatch.
Package assertion defines the ability for users to include assertions on the regular expression matched in ReelMatch.
tnf/handlers/generic/condition
Package condition defines the Condition Evaluate contract.
Package condition defines the Condition Evaluate contract.
tnf/handlers/generic/condition/intcondition
Package intcondition exposes some common integer condition implementations.
Package intcondition exposes some common integer condition implementations.
tnf/handlers/generic/condition/stringcondition
Package stringcondition exposes some common string condition implementations.
Package stringcondition exposes some common string condition implementations.
tnf/handlers/hostname
Package hostname provides a hostname discovery test utilizing the `hostname` Unix command.
Package hostname provides a hostname discovery test utilizing the `hostname` Unix command.
tnf/handlers/ipaddr
Package ipaddr provides an ip address polling test utilizing the `ip` Unix command.
Package ipaddr provides an ip address polling test utilizing the `ip` Unix command.
tnf/handlers/node
Package node provides a tnf.Test implementation for getting OpenShift node information for a cluster.
Package node provides a tnf.Test implementation for getting OpenShift node information for a cluster.
tnf/handlers/operator
Package operator provides functionality to run tests on operators.
Package operator provides functionality to run tests on operators.
tnf/handlers/ping
Package ping provides a test ICMPv4 implementation using the `ping` Unix command.
Package ping provides a test ICMPv4 implementation using the `ping` Unix command.
tnf/identifier
Package identifier provides a test identifier definition for tnf.Test implementations and a catalog of known tests.
Package identifier provides a test identifier definition for tnf.Test implementations and a catalog of known tests.
tnf/interactive
Package interactive provides common implementations of the expect.Expecter interface including oc, shell, and ssh.
Package interactive provides common implementations of the expect.Expecter interface including oc, shell, and ssh.
tnf/interactive/mocks
Package mock_expect is a generated GoMock package.
Package mock_expect is a generated GoMock package.
tnf/reel
Package reel runs a target subprocess with programmatic control over interaction with it.
Package reel runs a target subprocess with programmatic control over interaction with it.
tnf/testcases
Package testcases defines various test case templates configurations.
Package testcases defines various test case templates configurations.
tnf/testcases/data/cnf
Package cnf defines various test case templates definitions.
Package cnf defines various test case templates definitions.
tnf/testcases/data/operator
Package operator defines various test case templates definitions.
Package operator defines various test case templates definitions.
Package suite provides the entrypoint for running ginkgo tests and producing a claim file.
Package suite provides the entrypoint for running ginkgo tests and producing a claim file.
container
Package container provides Ginkgo tests used to validate container CNF facets.
Package container provides Ginkgo tests used to validate container CNF facets.
generic
Package generic contains generally applicable tests, and utilities for managing their configuration.
Package generic contains generally applicable tests, and utilities for managing their configuration.
operator
Package operator provides Ginkgo tests used to validate operator CNF facets.
Package operator provides Ginkgo tests used to validate operator CNF facets.
internal
api
Package api provides client to make rest api call to RH catalog and container published image database.
Package api provides client to make rest api call to RH catalog and container published image database.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL