In this first blog post of our series on Atlassian tools and 3rd-party plugins for Atlassian tools, we’ll have a close look on how to do test management in JIRA. You will get an insight in first thoughts and ideas when organizing tests and which main requirements are the bottom-line for test management in JIRA. After this we will show which types of test management plugins are available for JIRA. We will have a closer look on the specific functions of two representative examples and compare them with each other. And finally we will give you some hints on how to find the right plugin for your tests.
mgm technology partners has been performing test management since the company exists. In projects with many different customers we encountered many different solutions for test management – and even more different definitions what test management is in the first place at all.
We define test management as creating and managing test cases, the collection of test executions and the test executions themselves with specific results and other required metadata. Test reports on manual or automated test executions will give you information about the current quality of your product. During test execution you find errors and bugs which you report (via JIRA) back to the development so that they can be fixed before you release your product. Effective QA measures ensure that your product works nearly flawless at the start of operational use.
At this point, we will focus on the above given definition for manual tests but without specific information on how to handle errors or bugs. You can also implement automatic tests in JIRA. Since the proceeding is strongly depending though on which automatic tests you want to perform and in which environment this will happen, we will not discuss this in detail.
Why JIRA for test management?
JIRA is a tool in which you usually manage the requirements and tasks for the development of your software including the bugs found during testing and in the field. If you also manage your tests in JIRA you have everything using one tool. Especially when working in agile teams, which are by definition interdisciplinary, it will be easy to let e.g. developers perform different test tasks which are managed by the same tool as the development is. Managing the test tasks in JIRA as well allows you to include them in your sprint. This ensures that QA will not be forgotten.
Another thing one should consider is that one can implement test management in JIRA without having to set up new infrastructure, new tools and training for one’s teams – which is usually time consuming and will lead to a lot of discussions and other overhead before you can finally start introducing the tools. When using JIRA you just need a new plugin and some short introduction to the added features as all basic JIRA functionalities are already known to everybody.
How to manage testing with JIRA?
After having provided arguments for choosing JIRA, let us now focus on how to manage testing with JIRA.
The basic requirements for test management almost look the same in different systems. We have reduced them down to four requirements which are at the basis of each test management tool – no matter where and how you finally perform the test:
The first thing you need is a repository of test cases which you can copy and execute as often as required. These test cases contain the test steps, test data and each step’s expected results.
Second, you need something to collect all tests you like to execute in a specific sprint, version or release in.
Third, you need the test executions which are based on the test cases. A test execution is the copy of the test case and is executed once for a test run with a specific result and other specific metadata, e.g. executor, execution date and so on.
Fourth, you need a set of different reports for test results of test executions and test collections and / or individual test cases contained in test executions.
In the following picture, the test repository and different test collections are shown. Single test cases from the repository will be copied as test executions within the single collections. In addition it is also displayed how bugs can be linked to single tests or whole collections.
1: Illustration of an abstract solution for the four requirements. (Click to enlarge)
The first idea coming to one’s mind is to create some new issue types and a new workflow, and to organize everything in a test management project and work without any additional plugin. But at a point in time you will find out that the traceability of your tests is not what you expected. You will hardly get an overview of link-paths across more than two levels. You will see which test case has which executions. But in order to see in which test collection each execution is, you will have to look at the execution itself and so on as you will only see the next linked level from each starting point, but not the whole path.
In addition, it will be difficult to get useful reports and quick answers to questions like:
For a specific test case: Show me the execution result over all test collections in which executions of this test case happened.
For a specific bug: Show me all related test collections, executions and requirements.
For a specific bug: Show me the number of affected executions.
For a specific test collection: Show me the progress and the fail ratio calculated on contained executions.
Furthermore, the management of test cases, collection and executions will become more difficult than expected.
Possible plugins
This is the time where you will start searching for plugins at the Atlassian Marketplace. If you search for “test management” you will find a lot of plugins. Most plugins belong to one of the following categories:
Plugins that provide test management with specific issue types.
Plugins that provide test management with specific objects to organize tests.
Plugins that provide interfaces to external test management tools.
We like to keep everything simple and deal only with one tool – JIRA. Therefore we do not consider the third plugin category. However, most plugins will give you the possibility to implement e.g. test automation on different levels later on.
There are many plugins, which are worth looking at, but for this post we chose the following plugins belonging to the first two categories:
TestFLO for Test Management by InTENSO IT Experts: test management with issue types
Zephyr for JIRA – test management by Zephry: test management with specific objects
They are typical examples for the two categories. Also we think that they are quite good plugins which will enable you to manage your tests in JIRA in a professional way. Both will give you good solutions for the basic requirements mentioned above.
Functionality
In the next sections of this post, we will have a closer look on each previously defined requirement and will give you an overview of how each plugin solves it. We will not describe specific actions to create e.g. test cases or test cycles but we will describe the concept behind each artifact.
At the end we will discuss the basic differences and consequences for you and your work.
Repository with Test Case Templates and Tests
One can create repositories consisting of test cases with both plugins. In TestFLO they are called “Test Case Templates”, in Zephyr “Tests”. With both plugins it is a special issue type provided by the plugin. We recommend creating a separate project for the test repository to ensure that one does not accidently execute an issue from the development repository.
In addition, it is also recommendable to define a specific workflow for your Test Case Template (TestFLO) / Test (Zephyr) so that you can track whether a test is still executable or obsolete or not finally defined.
In a Test Case Template (TestFLO) / Test (Zephyr) the individual test steps are managed and all the executions which are based on this test definition are shown (see screen shots below).
2: TestFLO – Test Case Template: Screenshot of an issue with exemplary content. (Click to enlarge)
3: Zephyr – Test: Screenshot of an issue with exemplary content. (Click to enlarge)
Test collections with Test Cycles and Test Plans
“Test Cycles” in Zephyr and “Test Plans” in TestFLO gather the execution of all Tests or Test Case Templates. They are created for specific sprints, versions, releases or whatever defines your test intervals. When starting a Test Cycle / Test Plan you will have to execute all tests which are part of it.
The test state is shown for each test which is part of the collection. The available states (i.e. passed, failed, unexecuted) can be configured in each plugin. You will also get a summary of the entire collection as shown in the screenshot below.
While in TestFLO the Test Plan is also a JIRA issue, in Zephyr the Test Cycle is a proprietary object and not a JIRA issue. Based on this fact, you can customize the available fields for a Test Plan in TestFLO in the same way as you do with any other issue type in JIRA. You can assign the Test Plan to a specific user, log work on it and perform any action you can also perform with a standard JIRA issue.
In Zephyr this is not possible for a Test Cycle. In a simple environment this will not be a big issue. But after some time and with more specific requirements for your test process it could become a major issue not to be able to adapt Test Cycles to your needs.
4: TestFLO – Test Plan: Screenshot of issue with exemplary content. (Click to enlarge)
5: Zephyr – Test Cycle: Screenshot of object with exemplary content. (Click to enlarge)
Executions with Test Cases and Executions
Here we face the same situation as with test collections. Executions in Zephyr are proprietary objects and Test Cases in TestFLO are JIRA issues. So the same possibilities and limitations apply.
While executing a test you can set a state for each test step and comment it if necessary in both plugins. In TestFLO you link the bugs found to the TestCases or TestPlans themselves while in Zephyr you can link a bug to a specific test step and to the execution. In Zephyr these two possibilities to link to a bug get reported differently.
6: TestFLO – Test Case: Screenshot of issue with exemplary content. (Click to enlarge)
7: Zephyr – Execution: Screenshot of object with exemplary content. (Click to enlarge)
Reports
Both plugins provide a wide range of reports, metrics and dashboard gadgets.
For example:
An overview of existing test cases and collections.
The state of execution of a collection in numbers and percentages or displayed in a diagram.
An overview in which collections a test case was performed and what was the respective result.
A summary of how many bugs for each execution or collection were created; and a link to the related test-step or execution from the bug’s side.
Plus many more.
Both tools will provide plenty of possibilities to track your tests and create appropriate reports.
While Zephyr provides some limited JQL Extensions to create filters and use them in JIRA standard dashboard gadgets, TestFLO provides full access to JIRA Queries due to using standard JIRA issues for its artifacts.
8: Example dashboards and reports from TestFLO (left) and Zephyr (right). (Click to enlarge)
Findings and Tips
As we have shown, both plugins provide almost everything one needs to manage one’s tests in JIRA for smaller projects. At first sight there seems to be hardly any difference between the two plugins compared. But in fact there is one big difference that influences everything…
While working with JIRA, you will notice that standard tasks like searching, logging work, adding comments and so on will always function via issues. Issues in fact are the basic object to work with in JIRA. There is nearly no action that is not related to or performed with issues. Searches, reports and dashboards provide plenty of options to display issues and information from issues or graphics based on information from issues right away. That is the reason why it is easier to understand and work with TestFLO than with Zephyr: TestFLO also works with issues, Zephyr does not.
If your testers are not familiar with the concept of JIRA it does not really make a difference for them in the end. Maybe at first, the reduced interface of Zephyr would be a benefit as normal JIRA issues are overwhelming on fields, information, menus and sections if you are not used to them. While in Zephyr test cycles and executions are well structured, the execution is easy to perform and you do not have to leave the interface to create a bug. But after some time, once your testers will have got used to work with JIRA, they will feel the limitations of Zephyr in comparison to TestFLO, especially with regard to managing test cycles and executions.
In case your testers are regular JIRA users anyway or got used to work with JIRA and therefore know how to work with issues, filters and gadgets they will like TestFLO as this plugin follows the basic concept of JIRA and the idea of issues. With it you get the full capability to customize your test case templates, test plans and test cases with additional custom fields, a specific workflow and work logs.
When starting to create dashboards and working with reports you will also find that TestFLO provides more possibilities as you can use standard JIRA gadgets and reports to get the information you like. Zephyr provides an extension for JQL called ZQL, but it also has its limitations and needs to be learned first.
If you work agile and like to plan your tests within your sprints, TestFLO is much more adept since you can put your test plans directly into your sprint as issues and handle them like any other issue in your sprint. It feels more natural to work this way in JIRA. Though Zephyr provides the functionality of planning test cycles in sprints within JIRA 7.x and this works quite well, too, it remains that a test cycle is not an issue. In this case you realize that once again.
Both tools provide the basic functionality one needs for test management in JIRA. But we think it is better to work with TestFLO as one has issues with all their benefits like custom-fields, workflows with transitions, work-log, searching, filtering and reporting. Keep in mind, too, that you can extend your JIRA installation with other plugins. As most plugins focus on issues, they also work with TestFLO while special interfaces are required for Zephyr. Some plugins will provide them but not all of them.
Zephyr is the better choice for you if you like to work with a simplified GUI without all the JIRA specific issue views, menus and so on.
Extensions, interfaces and other useful information
Both plugins have some extra plugins which provide extensions and additional interfaces. We will not have a closer look on them. But we will give you an overview and short description of each extension since this could be important for later use of one or the other plugin.
Zephyr extensions and interfaces
Zephyr Blueprints for Confluence
This plugin provides blueprints for Confluence giving you the possibility to create a test dashboard in your Wiki or to add reports to test plans or protocols within Confluence.
ZAPI
This plugin extents the REST API to provide access to Zephyr artifacts programmatically.
Zephyr for JIRA Add-on Bamboo
Allows synchronizing test cases between JIRA and Bamboo. Here you can publish test results to JIRA in real-time for test cases executed in Bamboo.
TestFLO extensions and interfaces
TestFLO Reports
Provides additional reports for TestFLO. Source code can be downloaded and adapted to your own needs.
TestFLO Automation for test execution
Allows you to trigger the execution of tests on your continuous integration server or local computer and to publish the results back in the corresponding test case in JIRA.
TestFLO REST API
TestFLO itself has an extension for the REST API by default, so no separate plugin is required.
Versions and compatibilities
We have evaluated both plugins on separate JIRA 6.4.11 instances including JIRA Agile. We used Zephyr version 2.6.2 and TestFLO version 4.2.1-j64.
We would like to mention, that TestFLO keeps older versions of its plugin that are not compatible with the latest JIRA version up to date. You can find updates for the plugin which are compatible to JIRA 6.3 – 6.3.15 from 2015-11-21 and ones compatible to JIRA 6.4 – 6.4.12 from 2016-01-19.
Zephyr offers this too: The last update for the version compatible to JIRA 6.0 – 6.4.12 was on 2015-12-22.
(c) 2016 mgm technology partners. This posting "Test Management with JIRA" is part of the mgm technology blog. The author of the posting is
Benjamin Weinheimer.
We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.