Is mutation testing useful in practice?

Question

Do you have any examples of real life applications of mutation testing? Does it work better than simple test coverage tools? Or is it useless?

What are the advantages/disadvantages of mutation testing in the real world?

Answer 1

I recently did some investigations on mutation testing. Results are here:

http://abeletsky.blogspot.com/2010/07/using-of-mutation-testing-in-real.html

In short: mutation testing could give some information about quality of source code and tests, but it is not something straighforward to use.

Answer 2

Mutation testing has helped me identify problems with test case assertions.

For example, when you get a report that says "no mutant has been killed by test case x", you take a look, and it turns out the assertion had been commented out.

According to this paper, developers at Google use Mutation testing as a complement to code-review and pull-request inspections. They seem happy about the results:

Developers have decided to redesign large chunks of code to make them testable just so a mutant could be killed, they have found bugs in complex logical expressions looking at mutants, they have decided to remove code with an equivalent mutant because they deemed it a premature optimization, they’ve claimed the mutant saved them hours of debugging and even production outages because no test cases were covering the logic under mutation properly. Mutation testing has been called one of the best improvements in the code review verification in years. While this feedback is hardly quantifiable, combined with the sheer number of thousands of developers willing to inspect surfaced mutants on their code changes makes a statement.

Answer 3

The usefulness of unit tests is no longer discussed. They are essential in conception of a quality application. But, how can we assess their relevance? A code coverage indicator up to 100% doesn’t mean the code is 100% tested. This is just a view of executed code during unit tests execution. Mutation testing will allow you to have more confidence in your tests.

This is a two step process:

1. Generate mutants.
2. Check that the mutations are found by the tests.

I wrote a entire article about this process, including some concrete cases.

Answer 4

I looked at mutation test some time ago as a method for checking the efficacy of my automated regression testing scripts. Basically, a number of these scripts had missing checkpoints, so while they were exercising the application being tested correctly, they weren't verifying the results against the baseline data. I found that a far simpler method than changing the code was to write another application to introduce modifications to a copy of the baseline, and re-run the tests against the modified baseline. In this scenario, any test that passed was either faulty or incomplete.

This is not genuine mutation testing, but a method that uses a similar paradigm to test the efficacy of test scripts. It is simple enough to implement, and IMO does a good job.

Answer 5

I known that this is a old question but recently Uncle Bob write a blog post very interesting about mutating testing that can help understand the usefully of this type of testing:

Uncle Bob mutating testing blog post

Answer 6

I've played around with pitest for a small, contrived application:

http://pitest.org/

It's a java tool that automates mutant generation. You can run it against your test suite and it'll generate HTML reports for you indicating how many mutants were killed. Seemed quite effective and didn't require much effort to set up. There are actually quite a few nice tools in the Java world for this sort of thing. See also:

http://www.eclemma.org/

For coverage.

---

I think the concepts behind mutation testing are sound. It's just a matter of tool support and awareness. You're fighting a tradeoff between the simplicity of traditional code coverage metrics and additional complexity of this technique - it really just comes down to tools. If you can generate the mutants, then it will help expose weaknesses in your test cases. Is it worth the marginal increase in effort over the testing you already do? With pitest, I did find it turning up test cases that seemed non-obvious.

Mutation testing is an angle of attack that's quite different from the unit/functional/integration testing methodologies.

1. You test your test suite - it's a meta-test of your whole testing program.
2. It inspires additional test cases you might not have otherwise considered.