Solid Java unit test automation? (JUnit/Hamcrest/…

问题:

Intent

I am looking for the following:

• A solid unit testing methodology
  1. What am I missing from my approach?
  2. What am I doing wrong?
  3. What am I doing which is unnecessary?
• A way to get as much as possible done automatically

Current environment

• Eclipse as IDE
• JUnit as a testing framework, integrated into Eclipse
• Hamcrest as a "matchers" library, for better assertion readability
• Google Guava for precondition validation

Current approach

Structure

• One test class per class to test
• Method testing grouped in static nested classes
• Test method naming to specify behaviour tested + expected result
• Expected exceptions specified by Java Annotation, not in method name

Methodology

• Watch out for null values
• Watch out for empty List<E>
• Watch out for empty String
• Watch out for empty arrays
• Watch out for object state invariants altered by code (post-conditions)
• Methods accept documented parameter types
• Boundary checks (e.g. Integer.MAX_VALUE, etc...)
• Documenting immutability through specific types (e.g. Google Guava ImmutableList<E>)
• ... is there a list for this? Examples of nice-to-have testing lists:
  • Things to check in database projects (e.g. CRUD, connectivity, logging, ...)
  • Things to check in multithreaded code
  • Things to check for EJBs
  • ... ?

Sample code

This is a contrived example to show some techniques.

---

MyPath.java

import static com.google.common.base.Preconditions.checkArgument;
import static com.google.common.base.Preconditions.checkNotNull;
import java.util.Arrays;
import com.google.common.collect.ImmutableList;
public class MyPath {
  public static final MyPath ROOT = MyPath.ofComponents("ROOT");
  public static final String SEPARATOR = "/";
  public static MyPath ofComponents(String... components) {
    checkNotNull(components);
    checkArgument(components.length > 0);
    checkArgument(!Arrays.asList(components).contains(""));
    return new MyPath(components);
  }
  private final ImmutableList<String> components;
  private MyPath(String[] components) {
    this.components = ImmutableList.copyOf(components);
  }
  public ImmutableList<String> getComponents() {
    return components;
  }
  @Override
  public String toString() {
    StringBuilder stringBuilder = new StringBuilder();
    for (String pathComponent : components) {
      stringBuilder.append("/" + pathComponent);
    }
    return stringBuilder.toString();
  }
}

---

MyPathTests.java

import static org.hamcrest.Matchers.is;
import static org.hamcrest.collection.IsCollectionWithSize.hasSize;
import static org.hamcrest.collection.IsEmptyCollection.empty;
import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
import static org.hamcrest.core.IsEqual.equalTo;
import static org.hamcrest.core.IsNot.not;
import static org.hamcrest.core.IsNull.notNullValue;
import static org.junit.Assert.assertThat;
import org.junit.Test;
import org.junit.experimental.runners.Enclosed;
import org.junit.runner.RunWith;
import com.google.common.base.Joiner;
@RunWith(Enclosed.class)
public class MyPathTests {
  public static class GetComponents {
    @Test
    public void componentsCorrespondToFactoryArguments() {
      String[] components = { "Test1", "Test2", "Test3" };
      MyPath myPath = MyPath.ofComponents(components);
      assertThat(myPath.getComponents(), contains(components));
    }
  }
  public static class OfComponents {
    @Test
    public void acceptsArrayOfComponents() {
      MyPath.ofComponents("Test1", "Test2", "Test3");
    }
    @Test
    public void acceptsSingleComponent() {
      MyPath.ofComponents("Test1");
    }
    @Test(expected = IllegalArgumentException.class)
    public void emptyStringVarArgsThrows() {
      MyPath.ofComponents(new String[] { });
    }
    @Test(expected = NullPointerException.class)
    public void nullStringVarArgsThrows() {
      MyPath.ofComponents((String[]) null);
    }
    @Test(expected = IllegalArgumentException.class)
    public void rejectsInterspersedEmptyComponents() {
      MyPath.ofComponents("Test1", "", "Test2");
    }
    @Test(expected = IllegalArgumentException.class)
    public void rejectsSingleEmptyComponent() {
      MyPath.ofComponents("");
    }
    @Test
    public void returnsNotNullValue() {
      assertThat(MyPath.ofComponents("Test"), is(notNullValue()));
    }
  }
  public static class Root {
    @Test
    public void hasComponents() {
      assertThat(MyPath.ROOT.getComponents(), is(not(empty())));
    }
    @Test
    public void hasExactlyOneComponent() {
      assertThat(MyPath.ROOT.getComponents(), hasSize(1));
    }
    @Test
    public void hasExactlyOneInboxComponent() {
      assertThat(MyPath.ROOT.getComponents(), contains("ROOT"));
    }
    @Test
    public void isNotNull() {
      assertThat(MyPath.ROOT, is(notNullValue()));
    }
    @Test
    public void toStringIsSlashSeparatedAbsolutePathToInbox() {
      assertThat(MyPath.ROOT.toString(), is(equalTo("/ROOT")));
    }
  }
  public static class ToString {
    @Test
    public void toStringIsSlashSeparatedPathOfComponents() {
      String[] components = { "Test1", "Test2", "Test3" };
      String expectedPath =
          MyPath.SEPARATOR + Joiner.on(MyPath.SEPARATOR).join(components);
      assertThat(MyPath.ofComponents(components).toString(),
          is(equalTo(expectedPath)));
    }
  }
  @Test
  public void testPathCreationFromComponents() {
    String[] pathComponentArguments = new String[] { "One", "Two", "Three" };
    MyPath myPath = MyPath.ofComponents(pathComponentArguments);
    assertThat(myPath.getComponents(), contains(pathComponentArguments));
  }
}

---

Question, phrased explicitly

• Is there a list of techniques to use to build a unit test? Something much more advanced than my oversimplified list above (e.g. check nulls, check boundaries, check expected exceptions, etc.) perhaps available in a book to buy or a URL to visit?

• Once I have a method that takes a certain type of parameters, can I get any Eclipse plug-in to generate a stub for my tests for me? Perhaps using a Java Annotation to specify metadata about the method and having the tool materialise the associated checks for me? (e.g. @MustBeLowerCase, @ShouldBeOfSize(n=3), ...)

I find it tedious and robot-like to have to remember all of these "QA tricks" and/or apply them, I find it error-prone to copy and paste and I find it not self-documenting when I code things as I do above. Admittedly, Hamcrest libraries go in the general direction of specialising types of tests (e.g. on String objects using RegEx, on File objects, etc) but obviously do not auto-generate any test stubs and do not reflect on the code and its properties and prepare a harness for me.

Help me make this better, please.

PS

Do not tell me that I am just presenting code which is a silly wrapper around the concept of creating a Path from a list of path steps supplied in a static factory method please, this is a totally made-up example but it shows a "few" cases of argument validation... If I included a much longer example, who would really read this post?

回答1:

1. Consider using ExpectedException instead of @Test(expected.... This is because if for example you expect a NullPointerException and your test throws this exception in your setup (before calling the method under test) your test will pass. With ExpectedException you put the expect immediately before the call to the method under test so there is no chance of this. Also, ExpectedException allows you to test the exception message which is helpful if you have two different IllegalArgumentExceptions that might be thrown and you need to check for the correct one.

2. Consider isolating your method under test from the setup and verify, this will ease in test review and maintenance. This is especially true when methods on the class under test are invoked as part of setup which can confuse which is the method under test. I use the following format:

   public void test() {
      //setup
      ...
   
      // test (usually only one line of code in this block)
      ...
   
      //verify
      ...
   }
   

3. Books to look at: Clean Code, JUnit In Action, Test Driven Development By Example

   Clean Code has an excellent section on testing

4. Most example I have seen (including what Eclipse autogenerates) have the method under test in the title of the test. This facilitates review and maintenance. For example: testOfComponents_nullCase. Your example is the first I have seen that uses the Enclosed to group methods by method under test, which is really nice. However, it adds some overhead as @Before and @After do not get shared between enclosed test classes.

5. I have not started using it, but Guava has a test library: guava-testlib. I have not had a chance to play with it but it seems to have some cool stuff. For example: NullPointerTest is quote:

• A test utility that verifies that your methods throw {@link * NullPointerException} or {@link UnsupportedOperationException} whenever any * of their parameters are null. To use it, you must first provide valid default * values for the parameter types used by the class.

Review: I realize the test above was just an example but since a constructive review might be helpful, here you go.

1. In testing getComponents, test the empty list case as well. Also, use IsIterableContainingInOrder.

2. In testing of ofComponents, it seems that it would make sense to call getComponents or toString to validate that it properly handled the various non-error cases. There should be a test where no argument is passed to ofComponents. I see that this is done with ofComponents( new String[]{}) but why not just do ofComponents()? Need a test where null is one of the values passed: ofComponents("blah", null, "blah2") since this will throw an NPE.

3. In testing ROOT, as has been pointed out before, I suggest calling ROOT.getComponents once and doing all three verifications on it. Also, ItIterableContainingInOrder does all three of not empty, size and contains. The is in the tests is extraineous (although it is linguistic) and I feel is not worth having (IMHO).

4. In testing toString, I feel it is very helpful to isolate the method under test. I would have written toStringIsSlashSeparatedPathOfComponents as follows. Notice that I do not use the constant from the class under test. This is because IMHO, ANY functional change to the class under test should cause the test to fail.

   @Test     
   public void toStringIsSlashSeparatedPathOfComponents() {       
      //setup 
      String[] components = { "Test1", "Test2", "Test3" };       
      String expectedPath =  "/" + Joiner.on("/").join(components);   
      MyPath path = MyPath.ofComponents(components)
   
      // test
      String value = path.toStrign();
   
      // verify
      assertThat(value, equalTo(expectedPath));   
   } 
   

5. Enclosed will not run any unit test that is not in an inner class. Therefore testPathCreationFromComponents would not be run.

Finally, use Test Driven Development. This will ensure that your tests are passing for the right reason and will fail as expected.

回答2:

I see you put a lot of effort to really test your classes. Good! :)

My comments/questions would be:

• what about mocking? you do not mention any tool for this
• it seems to me that you care a lot about the nitty-gritty details (I do not say they are not important!), while neglecting the business purpose of tested class. I guess it comes from the fact you code code-first (do you?). What I would suggest is more TDD/BDD approach and focus on business responsibilities of the tested class.
• not sure what this gives you: "Method testing grouped in static nested classes"?
• regarding auto-generation of test stubs etc. Simply put: don't. You will end up testing implementation instead of behaviour.

回答3:

Ok, here is my view on your questions:

Is there a list of techniques to use to build a unit test?

Short answer, no. Your problem is that to generate a test for a method, you need to analyse what it does and put a test in for each possible value in each place. There are/were test generators, but IIRC, they didn't generate maintainable code (see Resources for Test Driven Development).

You've already got a fairly good list of things to check, to which I would add:

• Make sure all paths through your methods are covered.
• Make sure all important functionality is covered by more than one test, I use Parameterized a lot for this.

One thing I find really useful to do is to ask what should this method be doing, as opposed to what does this method do. This way, you write the tests with a more open mind.

Another thing I find useful is to cut down on the boilerplate associated with the tests, so I can read the tests more easily. The easier it is to add tests, the better. I find Parameterized very good for this. For me, readability of tests is key.

So, taking your example above, if we drop the requirement 'test only one thing in a method' we get

public static class Root {
  @Test
  public void testROOT() {
    assertThat("hasComponents", MyPath.ROOT.getComponents(), is(not(empty())));
    assertThat("hasExactlyOneComponent", MyPath.ROOT.getComponents(), hasSize(1));
    assertThat("hasExactlyOneInboxComponent", MyPath.ROOT.getComponents(), contains("ROOT"));
    assertThat("isNotNull", MyPath.ROOT, is(notNullValue()));
    assertThat("toStringIsSlashSeparatedAbsolutePathToInbox", MyPath.ROOT.toString(), is(equalTo("/ROOT")));
  }
}

I've done two things, I've added the description into the assert, and I've merged all of the tests into one. Now, we can read the test and see that we've actually got duplicate tests. We probably don't need to test is(not(empty()) && is(notNullValue()), etc. This violates the one assert per method rule, but I think it's justified because you've removed lots of boilerplate without cutting down on coverage.

Can I perform checks automatically?

Yes. But I wouldn't use annotations to do it. Let's say we have a method like:

public boolean validate(Foobar foobar) {
  return !foobar.getBar().length > 40;
} 

So I have a test method which says something like:

private Foobar getFoobar(int length) {
  Foobar foobar = new Foobar();
  foobar.setBar(StringUtils.rightPad("", length, "x")); // make string of length characters
  return foobar;
}

@Test
public void testFoobar() {
  assertEquals(true, getFoobar(39));
  assertEquals(true, getFoobar(40));
  assertEquals(false, getFoobar(41));
}

The above method is easy enough to factor out depending upon the length, into a Parameterized test of course. Moral of the story, you can factorize your tests just as you can with non-test code.

So to answer your question, in my experience, I've come to the conclusion that you can do a lot to help with all of the combinations by cutting down on boilerplate within your tests, by using a judicious combination of Parameterized and factorization of your tests. As a final example, this is how I would implement your test with Parameterized:

@RunWith(Parameterized.class) public static class OfComponents { @Parameters public static Collection data() { return Arrays.asList(new Object[][] { { new String[] {"Test1", "Test2", "Test3"}, null }, { new String[] {"Test1"}, null }, { null, NullPointerException.class }, { new String[] {"Test1", "", "Test2"}, IllegalArgumentException }, }); }

private String[] components;

@Rule
public TestRule expectedExceptionRule = ExpectedException.none();

public OfComponents(String[] components, Exception expectedException) {
   this.components = components;
   if (expectedException != null) {
     expectedExceptionRule.expect(expectedException);
   }
}

@Test
public void test() {
  MyPath.ofComponents(components);
}

Please note that the above isn't tested and probably doesn't compile. From the above, you can analyse the data as input and add (or at least think about adding) all of the combinations of everything. For instance, you haven't got a test for {"Test1", null, "Test2"} ...

回答4:

Well, I will post 2 difference answers.

1. As James Coplien stated unit test is worthless. I disagree with him on this issue, but may be you will find this helpful to consider to unit test less instead of searching for automatic solution.

2. Consider to use Theories with DataPoints. I think this will minimize your problem significantly. Also, using mock can help you.