Coverage vs reachable code

Question

Q: how to detect real test coverage ?

I've noticed one problem with code coverage metric and test quality: 100% code coverage doesn't mean that code is really tested.

Sometimes test gives 100% coverage even that it doesn't cover everything. Problem lays in coverage definition, we assume coverage==reachable code.

But it's not true, code could be 100% reachable but not 100% covered with the test.

Take a look into example, this test gives 100% coverage (EMMA), but in reality it doesn't cover values which will be passed to service mock. So, if value will be changed, test won't fail.

Example:

public class User {
  public static final int INT_VALUE = 1;
  public static final boolean BOOLEAN_VALUE = false;
  public static final String STRING_VALUE = "";
  private Service service;

  public void setService(Service service) {
      this.service = service;
  }

  public String userMethod() {
      return service.doSomething(INT_VALUE, BOOLEAN_VALUE, STRING_VALUE);
  }
}

And test for it:

public class UserTest {

  private User user;
  private Service easyMockNiceMock;

  @Before
  public void setUp() throws Exception {
      user = new User();
      easyMockNiceMock = EasyMock.createNiceMock(Service.class);
  }

  @Test
  public void nonCoverage() throws Exception {
      // given
      user.setService(easyMockNiceMock);
      expect(easyMockNiceMock.doSomething(anyInt(), anyBoolean(), (String) anyObject())).andReturn("");
      replay(easyMockNiceMock);
      // when
      user.userMethod();
      // then
      verify(easyMockNiceMock);
  }
}

Answer 1

Take a look at Jester, which performs mutation testing. From the site:

Jester finds code that is not covered by tests. Jester makes some change to your code, runs your tests, and if the tests pass Jester displays a message saying what it changed. Jester includes a script for generating web pages that show the changes made that did not cause the tests to fail.

Jester is different than code coverage tools, because it can find code that is executed by the running of tests but not actually tested. Jester's approach is called mutation testing or automated error seeding. However, Jester is not meant as a replacement for code coverage tools, merely as a complementary approach.

Answer 2

100% coverage has never meant 100% tested, and anyone who claimed it did either doesn't understand, or is lying to you. Coverage measurement simply measures what product code has been executed during testing. There are dozens of ways to write tests that produce 100% coverage, and then don't fully test your code.

The simplest way is to write a test that calls the product function, and then never makes any assertions about the return value!

Here's a blog post I wrote about this very topic: Flaws in Coverage Measurement, it's Python-centric, but the concepts are all the same.