Openstack projects use tox to manage virtual environments and run unit tests which I talked about here.
In this example I am using the oslo.config repo to look at the various tox configs in openstack use. The Governance Project Testing Interface is a starting point to read about project guidelines.
Get the current codebase
The last line helps me identify the specific git commit I am working with. When moving between branches or when looking at a repo that may be a few days old, if I need to recreate this exact codebase all I need is this. For example, to look at a prior version at 3ab403925e9cb2928ba8e893c4d0f4a6f4b27d7 for example.
To revert back to the current repo master.
NOTE: You don’t need to specify the full commit hash. In this example 3ab4039 also works.
tox configuration
The tox.ini file contains various config sections.
[tox] are global options
[testenv] are the default options for each virtual environment
[testenv:NAME] are the specific test environments
tox.ini
NOTE: These file differ between projects. See later for an example comparison with python-openstackclient, nova and horizon.
Prerequisites
The default [testenv] options first refer to requirements.txt and test-requirements.txt which define the specific packages and required versions. Either minimum (e.g. netaddr>=0.7.12), range (e.g. stevedore>=1.3.0,=0.6,!=0.7,
requirements.txt
Style Guidelines (PEP8)
The first test we look at is pep8 run by flake8. This starts by reviewing the code with Style Guide for Python Code and any specific Openstack Style Guidelines.
As with all unit tests you are wanting to see "The bar is green, the code is clean". An example of a failing test would look like:
Running tests
To run all tests for a given Python version you just specify said version.
You can pass a specific test or tests via command line identifying the names by looking at the test classes.
NOTE: This project has a top level /tests directory which uses the old import API and I am informed is being removed for liberty.
A failing test is going to produce the following.
Testr
This is a wrapper for the underlying testr command (found in the command line of the [testenv] section). We can reproduce what this runs manually with.
The current tox.ini config includes the --slowest argument which is self explaining.
One benefit of running this specifically is when writing failing tests (i.e. the Test Driven Development (TDD) approach to agile software development). You do not really want to run all tests in order to see a failure. The -f option helps.
NOTE: It takes a bit to realize the syntax of tox and testr and handling doubledash? -- placement. When you work it out you realize you can reproduce this with tox directly using:
The reason for dropping into an activated virtual environment and running testr manually is because tox will destroy and recreate your virtual environment each time the command is executed, which is time consuming.
The Testr source can be found at testrepository, identified by (py27)$ more `which testr`.
Testr syntax
Testr has multiple options and commands you can read about via various help options:
While debugging several testr commands were useful.
List all tests
1183 - 1 corresponds to the 1182 test run.
Last run
This enables you to review the last run tests (in a separate thread) and also get a correct error response code.
Code Coverage
The tox.ini also provides a section for code coverage.
Which is a wrapper for:
These commands produces a /cover directory (which is not currently in .gitignore). The contents are HTML. I suspect there is likely an option for a more CLI readable format however for simplicity we publish these to an available running web server.
Apache Setup
In order to view what code coverage produces I configured Apache with a separate port and vhost in this devstack environment.
Then I simply copied the projects coverage output as a quick hack to view.
Documentation
The last testenv setup in oslo.config is for documentation.
This creates a /doc directory (in .gitignore) which I copied to my previously configured web container to view in HTML.
Other tox.ini configuration
As I navigate around other Openstack projects I have noticed some differences. These include:
Alternative global settings
More detailed [testenv]
Some fancy output coloring.
Alternative [testenv:NAME] sections
Different Style guidelines
Different Code Coverage
Different Docs
Additional sections
What's Next
In a followup blog I will be talking about debugging with pdb and how to use this with tox.
References
Running openstack tests with tox
Installing Python 3.3 on Ubuntu 14.04.2 LTS
OpenStack Testr docs
tox reference documentation