Mutation Testing

Recently I came to the realisation that 100% code coverage by unit tests, while desirable, does not guarantee that anyone is checking the output of the code being called. While thinking about this, I ran across the concept of mutation testing, whereby you break your code, one line at a time, and make sure that doing so causes a test to fail. If it doesn’t, you don’t have enough tests.

Jester appears to be the dominant mutation testing tool. It is tightly coupled with the Java language and JUnit. There is a port to C# and NUnit, called Nester, but it is alpha-release and has been since 2004. This MSDN post explains how to do mutation testing on the IL directly, and comes with some fairly hairy code (VS 2003) to do so.

I started playing around with this, writing a small, buggy DLL, and equally small test jig, looking at the IL output by ildasm, and changing stuff at random. It doesn’t help that the IL output of this code does not include any of the 12 operators discussed in James McCaffrey’s post. But with the help of the IL spec I was able to change some code. And every change broke a test. But my code is still wrong!

My unit tests have 100% code coverage, and any changes to the inequality operators cause a test to fail. Is this just a limitation of the unit test, code coverage and mutation testing approaches? Is there some mutation I’m missing that will break my code but let my tests pass?

    public class MyMath 
        public int Max(int value1, int value2, int value3) 
            if (value2 < value3) 
                return value3; 
            if (value1 < value2) 
                return value2; 
            return value1; 

    public class UnitTest1 
        public void TestMethod1() 
            MyMath a = new MyMath(); 
            Assert.AreEqual(3, a.Max(1, 2, 3)); 
            Assert.AreEqual(3, a.Max(1, 3, 2)); 
            Assert.AreEqual(3, a.Max(3, 2, 1)); 
            // above three tests pass with 100% code coverage 

            //Assert.AreEqual(3, a.Max(3, 1, 2)); 
            // this test fails 

One Response to “Mutation Testing”

  1. Deinotes Says:

    Thinking about this some more, the problem is that the code correctly and minimally implements the contract defined by the tests. No amount of mutating the code is going to find anything, because the code, according to the contract defined by the tests, is correct. The only reason we think it’s wrong is because we think we know what Max() should do.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: