alvinashcraft
shared this story
from 
Jenkins Blog.
This is a guest post by Liam Newman,
Technical Evangelist at CloudBees.
The
JUnit plugin
is the go-to test result reporter for many Jenkins projects,
but the it is not the only one available. The
xUnit plugin
is a viable alternative that supports JUnit and many other test result file formats.
Introduction
No matter the project, you need to gather and report test results.
JUnit is one of the most widely supported formats for recording test results.
For a scenarios where your tests are stable and your framework can produce JUnit output,
this makes the JUnit plugin ideal for reporting results in Jenkins.
It will consume results from a specified file or path, create a report,
and if it finds test failures it will set the the job state to "unstable" or "failed".
There are also plenty of scenarios where the JUnit plugin is not enough.
If your project has some failing tests that will take some time to fix,
or if there are some flaky tests,
the JUnit plugin’s simplistic view of test failures may be difficult to work with.
No problem, the Jenkins plugin model lets us replace the JUnit
plugin functionality with similar
functionality from another plugin and Jenkins Pipeline lets us do this in safe
stepwise fashion where we can test and debug each of our changes.
In this article, I will show you how to replace the JUnit plugin with the
xUnit plugin in Pipeline code to address a few common test reporting scenarios.
Initial Setup
I’m going to use the "JS-Nightwatch.js" sample project from my
previous post to demonstrate a couple
common scenarios that the xUnit handles better.
I already have the latest
JUnit plugin
and
xUnit plugin
installed on my Jenkins server.
I’ll be keeping my changes in
link:my fork
of the "JS-Nightwatch.js" sample project on GitHub, under the
"blog/xunit" branch.
Here’s what the Jenkinsfile looked like at the end of that previous post and what
the report page looks like after a few runs:
Jenkinsfile
Switching from JUnit to xUnit
I’ll start by replacing JUnit with xUnit in my pipeline.
I use the Snippet Generator to create the step with the right parameters.
The main downside of using the xUnit plugin is that while it is Pipeline compatible,
it still uses the more verbose step() syntax and has some very rough edges around that, too.
I’ve filed
JENKINS-37611
but in the meanwhile, we’ll work with what we have.
If I replace the junit step in my Jenkinsfile with that last step above,
it produces a report and job result identical to the JUnit plugin but using the xUnit plugin. Easy!
Accept a Baseline
Most projects don’t start off with automated tests passing or even running.
They start with a people hacking and prototyping, and eventually they start to write tests.
As new tests are written, having tests checked-in, running, and failing can be valuable information.
With the xUnit plugin we can accept a baseline of failed cases and drive that number down over time.
I’ll start by changing the Jenkinsfile to fail jobs only if the number of failures is greater than an expected baseline,
in this case four failures. When I run the job with this change, the reported numbers remain the same, but the job passes.
Jenkinsfile
Next, I can also check that the plugin reports the job as failed if more failures occur.
Since this is sample code, I’ll do this by adding another failing test and checking the job
reports as failed.
tests/guineaPig.js
In a real project, we’d make fixes over a number of commits bringing the number of failures down and adjusting our baseline.
Since this is a sample, I’ll just make all tests pass and set the job failure threshold for failed and skipped cases to zero.
Jenkinsfile
tests/guineaPig.js
tests/guineaPig_1.js
Allow for Flakiness
We’ve all known the frustration of having one flaky test that fails once every ten jobs.
You want to keep it active so you can working isolating the source of the problem,
but you also don’t want to destablize your CI pipeline or reject commits that are actually okay.
You could move the test to a separate job that runs the "flaky" tests,
but in my experience that just leads to a job that is always in a failed state
and a pile of flaky tests no one looks at.
With the xUnit plugin, we can keep the this flaky test in main test suite but allow
the our job to still pass.
I’ll start by adding a sample flaky test. After a few runs, we can see the test
fails intermittently and causes the job to fail too.
tests/guineaPigFlaky.js
I can almost hear my teammates screaming in frustration just looking at this report.
To allow specific tests to be unstable but not others,
I’m going to add a guard "suite completed" test to the suites that should be stable,
and keep flaky test on it’s own.
Then I’ll tell xUnit to allow for a number of failed tests, but no skipped ones.
If any test fails other than the ones I allow to be flaky,
it will also result in one or more skipped tests and will fail the build.
tests/guineaPig.js
tests/guineaPig_1.js
After a few more runs, you can see the flaky test is still being flaky,
but it is no longer failing the build. Meanwhile, if another test fails,
it will cause the "suite completed" test to be skipped, failing the job.
If this were a real project, the test owner could instrument and eventually fix
the test. When they were confident they had stabilized the test the could add
a "suite completed" test after it to enforce it passing without changes to other
tests or framework.
Conclusion
This post has shown how to migrate from the JUnit plugin to the
xUnit plugin on an existing project in Jenkins pipeline. It also covered how to
use the features of xUnit plugin to get more meaningful and effective Jenkins
reporting behavior.
What I didn’t show was how many other formats xUnit supports - from CCPUnit to MSTest. You can
also write your own XSL for result formats not on the known/supported list.
Links
xUnit plugin
bitwiseman/JS-Nightwatch.js
saucelabs-sample-test-frameworks
