Archive for March, 2007

Mutation Testing

March 30, 2007

Recently I came to the realisation that 100% code coverage by unit tests, while desirable, does not guarantee that anyone is checking the output of the code being called. While thinking about this, I ran across the concept of mutation testing, whereby you break your code, one line at a time, and make sure that doing so causes a test to fail. If it doesn’t, you don’t have enough tests.

Jester appears to be the dominant mutation testing tool. It is tightly coupled with the Java language and JUnit. There is a port to C# and NUnit, called Nester, but it is alpha-release and has been since 2004. This MSDN post explains how to do mutation testing on the IL directly, and comes with some fairly hairy code (VS 2003) to do so.

I started playing around with this, writing a small, buggy DLL, and equally small test jig, looking at the IL output by ildasm, and changing stuff at random. It doesn’t help that the IL output of this code does not include any of the 12 operators discussed in James McCaffrey’s post. But with the help of the IL spec I was able to change some code. And every change broke a test. But my code is still wrong!

My unit tests have 100% code coverage, and any changes to the inequality operators cause a test to fail. Is this just a limitation of the unit test, code coverage and mutation testing approaches? Is there some mutation I’m missing that will break my code but let my tests pass?

    public class MyMath 
        public int Max(int value1, int value2, int value3) 
            if (value2 < value3) 
                return value3; 
            if (value1 < value2) 
                return value2; 
            return value1; 

    public class UnitTest1 
        public void TestMethod1() 
            MyMath a = new MyMath(); 
            Assert.AreEqual(3, a.Max(1, 2, 3)); 
            Assert.AreEqual(3, a.Max(1, 3, 2)); 
            Assert.AreEqual(3, a.Max(3, 2, 1)); 
            // above three tests pass with 100% code coverage 

            //Assert.AreEqual(3, a.Max(3, 1, 2)); 
            // this test fails