I bet that you wonder every day how to improve your testing and find issues before the software gets intI bet that you wonder every day how to improve your testing. You also wish to find issues before the software gets into production.
One of the great ideas of trying something new is to transform your tests scripts into real mutants 🙂
What is mutation testing?
Mutation testing is conceptually quite simple.
Faults (or mutations) are automatically seeded into the code then the tests are run. If tests fail, then the mutation is killed, but when the tests pass then the mutation survives. 😉
The quality of the automation test scripts can be gauged from the percentage of killed mutations. Fun!
A mutator is a change in the program logic, e.g. when it comes to conditionals: The “Negate Conditionals Mutator” changes conditional expressions into their opposite. An expression like
if (a == b) {
// do something
}
will be mutated to
if (a != b) {
// do something
}
What?
In one of my projects, we use PiTest (or PIT) as a tool for mutation testing.
It is an open-source framework for mutation testing in Java, so everyone can try it out.
PIT runs your unit tests against automatically modified versions of the application code. When the application code changes, it should produce different results and cause the unit tests to fail. On the other hand, if a unit test does not fail in this situation, it may indicate an issue with the test suite.
Why?
Traditional test coverage (i.e line, statement, branch, etc.) measures only which code is executed by your tests. It does not check if your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested.
The most extreme examples of the problem are tests with no assertions. Fortunately, these are uncommon in most code bases. Much more common is code that is only partially tested by its suite. A suite that only partially tests code can still execute all its branches (examples).
As it is actually able to detect whether each statement is meaningfully tested, mutation testing is the gold standard against which all other types of coverage are measured.
For sure, PiTest is not the only tool for mutation testing, but it is fast and reliable, so I can recommend it for your project. What is more, it is open-source, but it is actively developed and supported, so it is quite likely that you will not be left alone with your problems with the tool.
PiTest works well with development toolings such as Maven or IntelliJ.
The reports produced by PIT are in an easy-to-read format combining line coverage and mutation coverage information. In addition, it provides valuable insights into the quality of tests and helps to efficiently identify test gaps that need to be closed.
Is it for everybody?
The use of unit tests is part of the common development approach. Teams are measuring code coverage (condition and branch) usually as a part of the Definition of Done. On the other hand, this measurement doesn’t check the quality of the tests. In extreme situations, it may happen that we just increase coverage without testing something meaningful.
This is where mutation testing kicks in.
The basic idea is that a good test should not work anymore when the underlying code is changed, “mutated”.
PITEST performs for each class in scope suitable mutations on bytecode-level and checks if the associated unit tests fail or not. It then reports out the cases where mutations survived the test, either due to the quality of the test or due to a missing test, and the cases where mutations made the tests fail. The result is a coverage report (one per module), which summarises mutation coverage and allows to drill down.
How to integrate PITest
Maven
PITest can be added as a plug-in to Maven
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>LATEST</version>
</plugin>
|
The Maven command line is
mvn
org.pitest:pitest-maven:mutationCoverage
|
More details on Maven configuration options can be found here: http://pitest.org/quickstart/maven/
With the above basic configuration, html-reports of the form shown above are created for each Maven module.
In the context of Halloween, I hope you’ll give a chance to the mutants and try to improve your testing.
In case of any comments stalk me on Twitter or comment down below. Cheers!
@Kinga Test - the Blog 
