applifecycle-backend-e2e

command module

v0.2.7 Latest Latest Go to latest Published: May 11, 2021 License: Apache-2.0 Imports: 12 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/open-cluster-management/applifecycle-backend-e2e

Links

Open Source Insights

README ¶

tags: redhat, test

Design

https://hackmd.io/d50Lam9hS2eM4w-PburJWA?view

Usage

applifecyce-backend-e2e is a web server which provides a few endpoints, to help you ease the test burden and serve some automation needs.

Usage of applifecycle-backend-e2e:
-cfg string
the path to clusters config files (default "default-kubeconfigs")
-data string
the path to clusters config files
-t int
timeout for running each expectation unit (default 15)
-v int
log level (default 1)

Run test server

To run the test server(at background), you can:

go get github.com/open-cluster-management/applifecycle-backend-e2e@v0.1.6

# with default test cases
applifecycle-backend-e2e -cfg cluster_config &

# with your own test cases defined at testdata
applifecycle-backend-e2e -cfg cluster_config -data testdata &

Note: your own testdata directory should have the same structure as the default one.

Note: the above is how the operator e2e test runs the test server Subscription e2e test

Query test server for running/checking tests

Once you start the test server, you can query test server via curl, or query it within your ginkgo suite, or via your browser.

For example, run it in your browser:

http://localhost:8765/run?id=chn-001

{
    "test_id": "chn-001",
    "name": "checked expectations",
    "run_status": "succeed",
    "error": "",
    "details": [
        {
            "test_id": "chn-001",
            "target_cluster": "hub",
            "desc": "should have a channel on hub",
            "apiversion": "apps.open-cluster-management.io/v1",
            "kind": "channel",
            "name": "git",
            "namespace": "ch-git",
            "matcher": "byname",
            "args": {}
        }
    ]
}

Similar idea applied to the other endpoints.

{
    "test_id": "helper",
    "name": "registered handler",
    "run_status": "",
    "error": "",
    "details": {
        "/clusters": "show all the registered cluster info",
        "/clusters?id=": "show all the registered cluster info by id",
        "/expectations": "show all the registered expectations",
        "/expectations?id=": "show all the registered expectation by id",
        "/results?id=": "show results by id",
        "/run/stage?id=": "run stage by id",
        "/run?id=": "run test case by id",
        "/stages": "show all the registered stages",
        "/stages?id=": "show all the registered stage by id",
        "/testcases": "show all the registered test cases",
        "/testcases?id=": "show all the registered test case by id"
    }
}

Expand the server

Define and integrate your own test case and expectation

Form Inputs

define app yamls

❯ cat default-e2e-test-data/testcases/release-operator-e2e.json
[
{
  "test_id": "release-001",
  "desc": "helmrelease install test: guestbook deploy",
  "urls": ["https://raw.githubusercontent.com/open-cluster-management/multicloud-operators-subscription-release/e2e-test-setup/examples/test-guestbook010.yaml"],
  "target_cluster": "hub"
}
]

define app expectations

❯ cat default-e2e-test-data/expectations/release-e2e.json
[
{
                "test_id": "release-001",
                "target_cluster": "hub",
                "desc": "should have a helmrelease on hub",
                "apiversion": "apps.open-cluster-management.io/v1",
                "kind": "HelmRelease",
                "name": "guestbook-010",
                "namespace": "helmrelease-test",
                "matcher": "byname",
                "args": {}
        },
{
                "test_id": "release-001",
                "target_cluster": "hub",
                "desc": "should have a frontend deployment on hub",
                "apiversion": "apps/v1",
                "kind": "deployment",
                "name": "guestbook-010-gbapp-frontend",
                "namespace": "helmrelease-test",
                "matcher": "byname",
                "args": {}
        },

etc.

Test the output and put your case into test server

You can test your new case by doing:(assuming you install the server)

start the server with your new test case locations

applifecycle-backend-e2e -cfg <your kubeconfig> -data default-e2e-test-data
run your new test cases http://localhost:8765/run?id=<new case ID>

You can check other parameters such as if the case is parsed correctly or not, by hitting different endpoints.
added your test case to binary data files by run make gen

You need to run make gen, which will put your testcase to the default binary data file. Doing so, the client of test server can your tests directly.
add your test case to the test server e2e-test suite, so travis of the test server can run it for you just add another entry to the following:

https://github.com/open-cluster-management/applifecycle-backend-e2e/blob/main/client/e2e_client/e2e_client_test.go
PR your changes, after the PR merge, you can tag it with semVer syntac(eg, v0.1.x), which will give us better dependency control when use your test cases

Once your test case is tagged, you can point to the specific tag in a repo and consume it.

Leverage the new test case in an operator

in our operator point to the new test server tag, by modify the run-e2e-tests.sh

run-e2e-tests.sh

adding the new test case id to the testIDs array

The following example would work on the channel, subscription and subscription-release repos for now.

for example:

https://github.com/open-cluster-management/multicloud-operators-subscription-release/blob/master/e2e/e2e_test.go

func TestE2ESuite(t *testing.T) {
    if err := isSeverUp(); err != nil {
        t.Fatal(err)
    }

    testIDs := []string{"release-001"}

    for _, tID := range testIDs {
        if err := runner(tID); err != nil {
            t.Fatal(err)
        }
    }

    t.Logf("helm release e2e tests %v passed", testIDs)
}

test the added case by run make e2e in your repo, if everything works fine, then PR the repo.

Leverage the new test case in canary

Canary integeration overview:

entry point: the canary env give us an entry point at:

https://github.com/open-cluster-management/canary-scripts/tree/2.2-integration/squad-tests

As you can see per squad per folder, in the folder, there's run_test.sh, which will run a given test image via docker run command.

For our team, the run_test.sh is defined as the following,

https://github.com/open-cluster-management/applifecycle-backend-e2e/blob/main/client/e2e_client/e2e_client_test.go

As the canary will only accept JUnit format output and the output needs to be in $root_dir/results.
work with entry point:
1. we mapped the out test result at docker level in the run_test.sh by --volume $root_dir/results:/opt/e2e/client/canary/results \
2. in the test server, we had the NewJUnitReporter to generate the result at the Junit format.
  
  https://github.com/open-cluster-management/applifecycle-backend-e2e/blob/main/client/canary/e2e_canary_hello_world_suite_test.go
```
func TestAppLifecycle_API_E2E(t *testing.T) {
    RegisterFailHandler(Fail)

    RunSpecsWithDefaultAndCustomReporters(t,
        "Applifecycle-API-Test",
        []Reporter{printer.NewlineReporter{}, reporters.NewJUnitReporter(JUnitResult)})
}
```
1. we set up specific tests for canary
  
  https://github.com/open-cluster-management/applifecycle-backend-e2e/blob/main/client/canary/e2e_canary_hello_world_test.go
I added KUBE_DIR as kubeconfig directroy for canary, for better docker command set up... not sure it's a valid argument... might need to get rid of this logic

Add your test case to canary

test your cases in your local env, generate the binary data, then you need to add your test case to

https://github.com/open-cluster-management/applifecycle-backend-e2e/blob/main/client/canary/e2e_canary_hello_world_test.go
then PR your changes
update the Dockerfile to the new tag/version

Dockerfile
make sure your change is merged to release-2.2 branch of test server.

Beyond a single operator

default-e2e-test-data
├── expectations
│   ├── channel-e2e.json
│   ├── release-e2e.json
│   └── subscription-e2e.json
├── stages
│   └── helm-release.json
└── testcases
    ├── channel-operator-e2e.json
    ├── release-operator-e2e.json
    └── subscription-operator-e2e.json