11. Dovetail as a Generic Test Framework

11.1. Overview

Dovetail is responsible for the technical realization of the OPNFV Verification Program (OVP) and other compliance verification projects within the scope of the Linux Foundation Networking (LFN) umbrella projects. Dovetail provides a generic framework for executing a specific set of test cases which define the scope of a given compliance verification program, such as OVP.

This document aims at introducing what Dovetail generic framework looks like and how to develop within this framework.

11.2. Introduction of Dovetail Framework

The following diagram illustrates Dovetail generic framework.

../../../_images/dovetail_generic_framework.png

In this diagram, there are 5 main parts, TestcaseFactory, TestRunnerFactory, CrawlerFactory, CheckerFactory and test case groups.

  • TestcaseFactory: For each project, there needs to create its own testcase class such as FunctestTestcase and OnapVtpTestcase. All these classes are based on class Testcase. There are already many functions in this base class which are mainly used to parse test case configuration files. If no other special requirements exist, it only needs to initialize these classes with different types. Otherwise, it needs to overload or add some functions.
  • TestRunnerFactory: Similar to TestcaseFactory, each project has its own test runner class. Dovetail supports 2 kinds of test runners, DockerRunner and ShellRunner. For projects based on Docker, it needs to create their own test runner classes such as FunctestRunner which inherit from class DockerRunner. For other projects that are based on Shell, it can use ShellRunner directly. Test case runners provide many functions to support test cases runs such as preparing test tool of each project, run all the commands defined by each test case and clean the environment.
  • Test case groups: Each group is composed of one project configuration file and a set of test cases belonging to this project. These groups are used as the input of test runners to provide information of projects and test cases. For ShellRunner, it only needs test case configuratons as the input.
  • CrawlerFactory: This is used to parse the results of test cases and record them with unified format. The original result data report by each project is different. So it needs to create different crawler classes for different projects to parse their results.
  • CheckerFactory: This is used to check the result data generated by crawler. Each project should have its own checker class due to the different requirements of different projects.

11.3. Development with Dovetail Framework

Everyone who is interested in developing Dovetail framework to integrate new upstream test cases will face one of the two following scenarios:

  • Adding test cases that belong to integrated projects: There are already some projects integrated in Dovetail. These projects are coming from OPNFV (Open Platform for NFV) and ONAP (Open Network Automation Platform) communities. It will be much easier to add new test cases that belong to these projects.
  • Adding test cases that not belong to integrated projects: The test cases may belong to other projects that haven’t been integrated into Dovetail yet. These projects could be in OPNFV, ONAP or other communities. This scenario is a little more complicated.

11.3.1. Test cases belonging to integrated projects

Dovetail framework already includes a large amount of test cases. All these test cases are implemented by upstream projects in OPNFV and ONAP. The upstream projects already integrated in Dovetail are Functest, Yardstick and Bottlenecks from OPNFV and VNF SDK and VVP from ONAP.

In order to add a test case belonging to one of these projects, there only need to add one test case configuration file which is in yaml format. Following is the introduction about how to use the file to add one new test case. Please refer to Dovetail test case github for all configuration files of all test cases.

---
Test case name in Dovetail:
  name: Test case name in Dovetail
  objective: Test case description
  validate:
    type: 'shell' or name of the project already integrated in Dovetail
    testcase: The original test case name called in the project that it is developed
    image_name: Name of the Docker image used to run this test
    pre_condition:
      - 'Commands needed to be executed before running this test'
      - 'e.g. cp src_file dest_file'
    cmds:
      - 'Commands used to run this test case'
    post_condition:
      - 'Commands needed to be executed after running this test case'
  report:
    source_archive_files:
      - test.log
    dest_archive_files:
      - path/to/archive/test.log
    check_results_files:
      - results.json
    portal_key_file: path/to/key/logs/xxx.log
    sub_testcase_list:
      - sub_test_1
      - sub_test_2
      - sub_test_3

This is the complete format of test case configuration file. Here are some detailed description for each of the configuration options.

  • Test case name in Dovetail: All test cases should be named as ‘xxx.yyy.zzz’. This is the alias in Dovetail and has no relationship with its name in its own project. The first part is used to identify the project where this test case come from (e.g. functest, onap-vtp). The second part is used to classify this test case according to test area (e.g. healthcheck, ha). Dovetail supports to run whole test cases in one test suite with the same test area. Also the area is used to group all test cases and generate the summary report at the end of the test. The last part is special for this test case itself (e.g. image, haproxy, csar). It’s better to keep the file name the same as the test case name to make it easier to find the config file according to this test case alias in Dovetail.
  • validate: This is the main section to define how to run this test case.
    • type: This is the type of this test case. It can be shell which means running this test case with Linux bash commands within Dovetail container. Also it can be one of the projects already integrated in Dovetail (functest, yardstick, bottlenecks, onap-vtp and onap-vvp). Then this type is used to map to its project configuration yaml file. For example, in order to add a test case in OPNFV project Functest to Dovetail framework, the type here should be functest, and will map to functest_config.yml for more configurations in project level. Please refer to Dovetail project config github for more details.
    • testcase: This is the name defined in its own project. One test case can be uniquely identified by type and testcase. Take the test case functest.vping.ssh as an example. Its type is ‘functest’ and testcase is ‘vping_ssh’. With these 2 properties, it can be uniquely identified. End users only need to know that there is a test case named functest.vping.ssh in OVP compliance test scope. Dovetail Framework will run vping_ssh within Functest Docker container.
    • image_name: [optional] If the type is shell, there is no need to give this. For other types, there are default docker images defined in their project configuration files. If this test case uses a different docker image, it needs to overwrite it by adding image_name here. The image_name here should only be the docker image name without tag. The tag is defined in project’s configuration file for all test cases belonging to this project.
    • pre_condition: [optional] A list of all preparations needed by this test case. If the list is the same as the default one in its project configuration file, then there is no need to repeat it here. Otherwise, it’s necessary to overwrite it. If its type is shell, then all commands in pre_condition, cmds and post_condition should be executable within Dovetail Ubuntu 14.04 Docker container. If its type is one of the Docker runner projects, then all commands should be executable within their own containers. For Functest, it’s alpine 3.8. For Yardstick and Bottlenecks it’s Ubuntu 16.04. For VNF SDK it’s Ubuntu 14.04. Also all these commands should not require network connection because some commercial platforms may be offline environments in private labs.
    • cmds: [optional] A list of all commands used to run this test case.
    • post_condition: [optional] A list of all commands needed after executing this test case such as some clean up operations.
  • report: This is the section for this test case to archive some log files and provide the result file for reporting PASS or FAIL.
    • source_archive_files: [optional] If there is no need to archive any files, this section can be removed. Otherwise, this is a list of all source files needed to be archived. All files generated by all integrated projects will be put under $DOVETAIL_HOME/results. In order to classify and avoid overwriting them, it needs to rename some important files or move them to new directories. Navigating directory $DOVETAIL_HOME/results to find out all files needed to be archived. The paths here should be relative ones according to $DOVETAIL_HOME/results.
    • dest_archive_files: [optional] This should be a list corresponding to the list of source_archive_files. Also all paths here should be relative ones according to $DOVETAIL_HOME/results.
    • check_results_files: This should be a list of relative paths of the result files generated by this test case. Dovetail will parse these files to get the result (PASS or FAIL).
    • portal_key_file: This should be the key log file of this test case which will be used by the OVP portal for review.
    • sub_testcase_list: [optional] This section is almost only for Tempest tests in Functest. Take functest.tempest.osinterop as an example. The sub_testcase_list here is an check list for this kind of tempest tests. Only when all sub test cases list here are passed, this test case can be taken as PASS. The other kind of tempest tests is tempest_custom such as functest.tempest.image. Besides taking the sub_testcase_list as the check list, it’s also used to generate an input file of Functest to define the list of sub test cases to be tested.

11.3.2. Test cases not belonging to integrated projects

If test cases waiting to be added into Dovetail do not belong to any project that is already integrated into Dovetail framework, then besides adding the test case configuration files introduced before, there are some other files needed to be added or modified.

11.3.2.1. Step 1: Add a project configuration file

For a new test case that belongs to a new project, it needs to create a project configuration file to define this new project in Dovetail first. Now Dovetail only supports integration of projects by using their Docker images. If this test case should be run with shell runner, then can only add test case configuration files with type ‘shell’ as describing before and skip the following steps. Following is the introduction of how to use project configuration file to add one new project into Dovetail. Please refer to Dovetail projects configuration github for all configuration files of all integrated projects.

---

{% set validate_testcase = validate_testcase or '' %}
{% set testcase = testcase or '' %}
{% set dovetail_home = dovetail_home or '' %}
{% set debug = debug or 'false' %}
{% set build_tag = build_tag or '' %}
{% set userconfig_dir = '/tmp/userconfig' %}
{% set patches_dir = '/tmp/patches' %}
{% set result_dir = '/tmp/results' %}
{% set openrc_file = '/home/conf/env_file' %}

project name:
  image_name: name of the docker image
  docker_tag: tag of the docker image
  opts:
    detach: true
    stdin_open: true
    privileged: true
  shell: '/bin/bash'
  envs:
    - 'CI_DEBUG={{debug}}'
    - 'DEPLOY_SCENARIO={{deploy_scenario}}'
    - 'ENV_NAME=env_value'
  volumes:
    - '{{dovetail_home}}/userconfig:{{userconfig_dir}}'
    - '{{dovetail_home}}/results:{{result_dir}}'
    - '/path/on/host:/path/in/container'
    - '/path/of/host/file:/file/path/in/container'
  mounts:
    - 'source={{dovetail_home}}/pre_config/env_config.sh,target={{openrc_file}}
    - 'source={{dovetail_home}}/pre_config,target=/home/opnfv/pre_config'
    - 'source=/file/or/derectory/on/host,target=/file/or/derectory/in/container'
  patches_dir: {{patches_dir}}
  pre_condition:
    - 'Commands needed to be executed before running this test'
  cmds:
    - 'Commands used to run this test case'
  post_condition:
    - 'Commands needed to be executed after running this test case'
  openrc: absolute path of openstack credential files
  extra_container:
    - container1_name
    - container2_name

This is the complete format of project configuration file. Here are some detailed description for each of the configuration options.

  • Jinja Template: At the begining of this yaml file, it uses Jinja template to define some parameters that will be used somewhere in this file (e.g. result_dir and openrc_file). Besides those, there are some other parameters providing by Dovetail framework as input of this file, and other parameters can be defined by using these ones (e.g. testcase and dovetail_home). The whole input parameters which can be used are list below.
    • attack_host: This is the attack host name of the test case which calls this project configuration file. It’s only for HA test cases and can be given in HA configuration file pod.yaml.
    • attack_process: This is the attack process name of the test case which calls this project configuration file. It’s only for HA test cases and can be given in HA configuration file pod.yaml.
    • build_tag: This is a string includes the UUID generated by Dovetail.
    • cacert: This is also only for OpenStack test cases. It is the absolute path of the OpenStack certificate provided in env_config.sh file.
    • deploy_scenario: This is the input when running Dovetail with option –deploy-scenario.
    • debug: This is True or False according to the command running test cases with or without option –debug.
    • dovetail_home: This is the DOVETAIL_HOME getting from the ENV.
    • os_insecure: This is only for test cases aiming at OpenStack. This is True or False according to env_config.sh file.
    • testcase: This is the name of the test case which calls this project configuration file. Different from validate_testcase, this is the alias defined in Dovetail not in its own project.
    • validate_testcase: This is the name of the test case instance which calls this project configuration file. The name is provided by the configuration file of this test case (validate -> testcase).
  • project name: This is the project name defined in Dovetail. For example OPNFV Functest project is named as ‘functest’ here in Dovetail. This project name will be used by test case configuration files as well as somewhere in Dovetail source code.
  • image_name: This is the name of the default Docker image for most test cases within this project. Each test case can overwrite it with its own configuration.
  • docker_tag: This is the tag of all Docker images for all test cases within this project. For each release, it should use one Docker image with a stable and official release version.
  • opts: Here are all options used to run Docker containers except ‘image’, ‘command’, ‘environment’, ‘volumes’, ‘mounts’ and ‘extra_hosts’. For example, the options include ‘detach’, ‘privileged’ and ‘tty’. The full list of all options can be found in Docker python SDK docs.
  • shell: This is the command used to run in the container.
  • envs: This is a list of all envs used to run Docker containers.
  • volumes: A volume mapping list used to run Docker containers. The source volumes list here are allowed to be nonexistent and Docker will create new directories for them on the host. Every project should at least map the $DOVETAIL_HOME/results in the test host to containers to collect all result files.
  • mounts: A mount mapping list used to run Docker containers. More powerful alternative to volumes. The source volumes list here are not allowed to be nonexistent. Every project should at least mount the $DOVETAIL_HOME/pre_config in the test host to containers to get config files.
  • patches_dir: [optional] This is an absolute path of the patches applied to the containers.
  • pre_condition: A list of all default preparations needed by this project. It can be overwritten by configurations of test cases.
  • cmds: A list of all default commands used to run all test cases within this project. Also it can be overwritten by configurations of test cases.
  • post_condition: A list of all default cleaning commands needed by this project.
  • openrc: [optional] If the system under test is OpenStack, then it needs to provide the absolute path here to copy the credential file in the Test Host to containers.
  • extra_container: [optional] The extra containers needed to be removed at the end of the test. These containers are created by the test cases themselves at runtime rather than created by Dovetail.

11.3.2.4. Step 4: Update unit tests

A patch is not going to be verified without 100% coverage when applying acceptance check.