Why do you need mutation testing?

I bet that you wonder every day how to improve your testing and found issues before the software gets into production.

One of the great ideas of trying something new is to transform your tests scripts into real mutants ๐Ÿ™‚

What is mutation testing?

Mutation testing is conceptually quite simple.

Faults (or mutations) are automatically seeded into the code, then the tests are run. If tests fail, then the mutation is killed, but when the tests pass then the mutation survives. ๐Ÿ˜‰

The quality of the automation test scripts can be gauged from the percentage of killed mutations.|Fun!

A mutator is a change in the program logic, e.g. when it comes to conditionals: The “Negate Conditionals Mutator” changes conditional expressions into their opposite. An expression like

if (a == b) {
  // do something

}

will be mutated to
if (a != b) {
  // do something

}

https://vignette.wikia.nocookie.net/despicableme/images/5/52/Screenshot_2016-02-10-01-09-16.jpg/revision/latest?cb=20161028002525

What?

In one of my projects we use PiTest (or PIT) as a tool for mutation testing.
It is an open-source framework for mutation testing in Java, so everyone can try it out.

PIT runs your unit tests against automatically modified versions of the application code. When the application code changes, it should produce different results and cause the unit tests to fail. On the other hand, if a unit test does not fail in this situation, it may indicate an issue with the test suite.

Why?

Traditional test coverage (i.e line, statement, branch, etc.) measures only which code is executed by your tests. It does not check if your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested.

The most extreme examples of the problem are tests with no assertions. Fortunately, these are uncommon in most code bases. Much more common is code that is only partially tested by its suite. A suite that only partially tests code can still execute all its branches (examples).

As it is actually able to detect whether each statement is meaningfully tested, mutation testing is the gold standard against which all other types of coverage are measured.

For sure, PiTest is not the only tool for mutation testing, but it is fast and reliable, so I can recommend it for your project. What is more, it is open-source, but it is actively developed and supported, so it is quite likely that you will not be left alone with your problems with the tool.

PiTest works well with development toolings such as Maven or IntelliJ.
The reports produced by PIT are in an easy to read format combining line coverage and mutation coverage information. In addition, it provides valuable insights into the quality of tests and helps to efficiently identify test gaps which need to be closed.

Is it for everybody?

Use of unit tests is part of common development approach. Teams are measuring code coverage (condition and branch) usually as a part of the Definition of Done. On the other hand, this measurement doesn’t check the quality of the tests. In extreme situations it may happen that we just increase coverage without testing something meaningful.

This is where mutation testing kicks in.

The basic idea is that a good test should not work anymore when the underlying code is changed, “mutated”. 

https://i.ytimg.com/vi/Z0-4tA6UCVE/maxresdefault.jpg

PITEST performs for each class in scope suitable mutations on bytecode-level and checks if the associated unit tests fail or not. It then reports out the cases where mutations survived the test, either due to quality of the test or due to a missing test, and the cases where mutations made the tests fail. The result is a coverage report (one per module), which summarises mutation coverage and allows to drill-down.

How to integrate PITest

Maven

PITest can be added as a plug-in to Maven

<plugin>     <groupId>org.pitest</groupId>     <artifactId>pitest-maven</artifactId>     <version>LATEST</version> </plugin>

The Maven command line is

mvn org.pitest:pitest-maven:mutationCoverage

More details on Maven configuration options can be found here: http://pitest.org/quickstart/maven/

With the above basic configuration, html-reports of the form shown above are created for each Maven module.

In the context of Halloween, I hope you’ll give a chance to the mutants and try to improve your testing.

In case of any comments stalk me on Twitter or comment down below. Cheers!

A screenshot of a social media post

Description automatically generated

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s