4. Disabling Strict API Validation in Tempest

4.1. Introduction

In 2015, the OpenStack QA team introduced a validation mechanism for Nova API responses in Tempest [1] with the goal of enforcing Nova API micro-versions. The API validation mechanism verifies that API responses only contain data elements (properties) as explicitly defined in API response schemas [2]. In case additional data elements are found in Nova API responses, the corresponding tests fail immediately without asserting whether or not the particular API operation actually succeeded or not.

Independently, cloud vendors have extended their commercial OpenStack cloud implementations with additional functionality which requires API extensions. Consequently, such cloud implementations do not pass Tempest tests which validate API responses despite actually implementing and providing the tested functionality.

This document describes an exemption process for use within the OPNFV Verification Program which

  1. allows vendors to pass Tempest tests if the tested functionality is fully supported despite the presence of additional data elements in API responses, and
  2. makes the application of the exemption process transparently visible in test results.

4.2. Background and benefits for OVP

Vendors of commercial NFV products have extended OpenStack to provide additional (NFV) functionality to their customers and to fill functional gaps in OpenStack. These add-ons potentially extend the OpenStack API in two ways:

  1. new API endpoints and
  2. additional attributes returned by existing API endpoints.

New API endpoints typically go unnoticed by OpenStack Tempest tests and hence do not interfere with existing tests. In contrast, (Nova) Tempest tests actively validate the responses returned by existing API endpoints against pre-defined schemas. An API response is considered invalid if additional attributes are present (see example below). Hence, this particular type of functional extension of OpenStack causes existing Tempest tests to fail, irrespective of whether or not the functionality which is supposed to be tested is actually available. As a result, a Tempest test failing due to extended API responses does not provide information about whether the tested functionality is available or not.

The OPNFV Verification Program has inherited the policy to strictly validate API responses from OpenStack by including a selection of Tempest tests in its compliance test suite. However, it was never discussed if OVP should adopt this policy as well. It turns out that this policy causes challenges for vendors of commercial NFV offerings to pass the OVP test suite. The exemption process outlined in this document aims at allowing to selectively disable strict API response validation in order to enable vendors to adopt OVP if the tested functionality is supported.

It must be clearly understood that this exemption targets only the scenario in which additional attributes are included in API responses. It does not provide a loophole for passing OVP tests if the OpenStack APIs have been altered significantly as this is in conflict with the objective of OVP to create industry alignment.

In conclusion, the exemption process described here is deemed beneficial for both the broader industry as well as for OVP: Enabling adoption of OVP by vendors which extended OpenStack API responses facilitates adoption of OVP in the industry. The limited validity period of an exemption incentivizes eventual alignment within the industry around a clearly specified set of APIs.

4.2.1. Example: additional attributes per VM for HA policy

This fictional example showcases the presence of an additional attribute in an API response. The example shows that the ‘server details’ [3], i.e. the VM metadata, includes an additional attribute ‘ha-policy’ which is used to associate high-availability policies with a VM instance. This attribute is utilized by a proprietary add-on component to manage VM migration and recovery in case of compute host failures:

{
  "server": {
    "accessIPv4": "1.2.3.4",
    "config_drive": "",
    "flavor": {...},
    "image": {...},
    "ha_policy": "migrate"       <-- additional attribute
    "name": "new-server-test",
    "status": "ACTIVE"
  }
}

4.3. Precedent in OpenStack

In the OpenStack community, the OpenStack Interoperability Working Group (Interop WG) [4] is maintaining multiple API interoperability compliance programs [5]. These programs utilize Tempest-based tests to determine if a given commercial cloud is compliant to a selected set of OpenStack APIs. After introduction of the strict API response validation, various cloud products which previously passed the compliance program failed validation because of the reasons outlined above.

In order to mitigate this situation, the Interop WG consulted with the broader OpenStack community [6] and eventually introduced an “additional properties waiver” for its API compliance programs in July 2016. The waiver was created with a clearly defined validity period, covering roughly one year - equivalent to three iterations of interoperability guidelines (2015.07, 2016.01, and 2016.08). The limited lifetime of the waiver was intended to give cloud product vendors a transition period for adapting their products to achieve full API compliance by the end of the exemption period. All details of the waiver are listed in [7]. Finally, the waiver was officially canceled in October 2017 [8] after about 15 months.

4.4. Exemption process for additional properties in API responses in the OVP

The details of the exemption process for disabling strict validation of API responses is as follows:

  1. The Dovetail tool provides a new command line option “–no-api-validation” for disabling strict API validation. This option needs to be explicitly given on the command line to disable strict API validation. If this command line option is omitted, the default behavior (i.e., strict API validation) is applied.
  2. The test results created by the Dovetail tool includes an explicit print-out stating if strict API validation was disabled during the test run or not.
  3. The OVP portal reads the uploaded result files and indicates for all uploaded test results if strict API validation was disabled or not.
  4. Together with the application for certification, a company can request an exemption from the requirement for strict API response checking. A rationale for requesting the exemption has to be provided. The request is either granted or rejected by the C&C committee. The rationale provided must establish that the need for exemption is not in violation of the OVP’s objectives.
  5. Compliance badges obtained under exemption are valid for one year.
  6. OPNFV expects OVP participants to aim for full compliance without requiring exemptions as soon as possible. Hence, an exemption can only be requested twice for the same product (addressing new versions of OVP or new versions of the product).
  7. The same logo will be used regardless of being obtained under exemption or not.
  8. The exemption will be made available to participants of OVP as part of a service release of OVP 2018.01, 2018.09 and 2019.12.
  9. The C&C committee will monitor the situation around exemptions and may decide changes to the above process at any time, including the possibility to stop issuing exemptions.
[1]https://review.opendev.org/gitweb?p=openstack%2Ftempest.git;a=commitdiff;h=f0c30bc241e5160e3fe7402e738ea8f56a8b1315
[2]https://github.com/openstack/tempest/tree/master/tempest/lib/api_schema/response/compute
[3]https://docs.openstack.org/api-ref/compute/#show-server-details
[4]https://wiki.openstack.org/wiki/Governance/InteropWG
[5]https://www.openstack.org/brand/interop/
[6]http://lists.openstack.org/pipermail/openstack-dev/2016-June/097349.html
[7]https://review.opendev.org/gitweb?p=openstack%2Finterop.git;a=commitdiff;h=c38e18b343505f16a74a97b748362fa7f1a01e57
[8]https://review.opendev.org/gitweb?p=openstack%2Finterop.git;a=commitdiff;h=5748c296a658cf5efebc16ad9d7644ca1125b073