8.1.4. Stress Test Specification

8.1.4.1. Scope

The stress test involves testing and verifying the ability of the SUT to withstand stress and other challenging factors. Main purpose behind the testing is to make sure the SUT is able to absorb failures while providing an acceptable level of service.

8.1.4.2. References

This test area references the following specifications, definitions and reviews:

8.1.4.3. Definitions and Abbreviations

The following terms and abbreviations are used in conjunction with this test area

  • iff - if and only if

  • NFVI - Network Functions Virtualization Infrastructure

  • NaN - Not a Number

  • Service Level - Measurable terms that describe the quality of the service provided by the SUT within a given time period

  • SUT - System Under Test

  • VM - Virtual Machine

8.1.4.4. System Under Test (SUT)

The system under test is assumed to be the NFVI and VIM in operation on a Pharos compliant infrastructure.

8.1.4.5. Test Area Structure

According to the testing goals stated in the test scope section, preceding test will not affect the subsequent test as long as the SUT is able to sustain the given stress while providing an acceptable level of service. Any FAIL result from a single test case will cause the SUT failing the whole test.

8.1.4.6. Test Descriptions

8.1.4.6.1. Test Case 1 - Concurrent capacity based on life-cycle ping test

8.1.4.6.1.1. Short name

bottlenecks.stress.ping

8.1.4.6.1.2. Use case specification

This test case verifies the ability of the SUT concurrently setting up VM pairs for different tenants (through different OpenStack related components) and providing acceptable capacity under stressful conditions. The connectivity between VMs in a VM pair for a tenant is validated through Ping test. A life-cycle event in this test case is particularly referred to a VM pair life-cycle consisting of spawning, pinging and destroying.

8.1.4.6.1.3. Test preconditions

  • heat_template_version: 2013-05-23

  • ElasticSearch Port: 9200

  • LogStash Port: 5044

  • Kibana Port: 5601

  • Yardstick Port: 5000

8.1.4.6.1.4. Basic test flow execution description and pass/fail criteria

8.1.4.6.1.4.1. Methodology for validating capacity of the SUT

Validating capacity of the SUT based on life-cycle ping test generally involves 2 subtests which provides secondary validation for the SUT furnishing users with reliable capacity without being crushed.

Let N1, N2, N3 and P1 be certain preset numbers, respectively. In subtest 1, the SUT concurrently setting up N1 VM pairs with each VM pair belonging to a different tenant. Then VM1 in a VM pair pings VM2 for P1 times with P1 packets. The connectivity could be validated iff VM1 successfully pings VM2 with these P1 packets. Subtest 1 is finished iff all the concurrent (N1) requests for creating VM pairs are fulfilled with returned values that indicate the statuses of the VM pairs creations.

Subtest 2 is executed after subtest 1 as secondary validation of the capacity. It follows the same workflow as subtest 1 does to set up N2 VM pairs.

Assume S1 and S2 be the numbers of VM pairs that are successfully created in subtest 1 and subtest 2, respectively. If min(S1,S2)>=N3, then the SUT is considered as PASS. Otherwise, we denote the SUT with FAIL.

Note that for subtest 1, if the number of successfully created VM pairs, i.e., S1, is smaller than N3. Subtest 2 will not be executed and SUT will be marked with FAIL.

8.1.4.6.1.4.2. Test execution

  • Test action 1: Install the testing tools by pulling and running the Bottlenecks Docker container

  • Test action 2: Prepare the test by sourcing openstack credential file, eliminating the environment constraints, i.e., Quota setting, setting up Yardstick docker, pulling and registering OS images and VM flavor

  • Test action 3: Call Yardstick to concurrently creating N1 VM pairs for N1 tenants

  • Test action 4: In each VM pair, VM1 pings VM2 for P1 times with P1 packets while recording the successful numbers

  • Test action 5: Mark the VM pairs with P1 successful pings as PASS and record the total number of PASS VM pairs as S1

  • Test action 6: Destroy all the VM pairs

  • Test action 7: If S1<N3, the SUT is marked with FAIL and the test return. Otherwise go to Test action 8

  • Test action 8: Go to Test action 3 and do the test again to create N2 VM pairs with PASS VM pairs counted as S2

  • Test action 9: If S2<N3, the SUT is marked with FAIL. Otherwise marked with PASS.

8.1.4.6.1.4.3. Pass / Fail criteria

Typical setting of (N1, N2, N3, P1) is (5, 5, 5, 10). The reference setting above is acquired based on the results from OPNFV CI jobs and testing over commercial products.

The connectivity within a VM pair is validated iff:

  • VM1 successfully pings VM2 for P1 times with P1 packets

The SUT is considered passing the test iff:

  • min(S1,S2)>=N3

Note that after each subtest, the program will check if the successfully created number of VM pairs is smaller than N3. If true, the program will return and the SUT will be marked with FAIL. Then the passing criteria is equal to the equation above. When subtest 1 returns, S2 here is denoted by NaN.

8.1.4.6.1.5. Post conditions

N/A