Integration Test of REST APIs with Code Coverage

问题:

We have built a REST API that exposes bunch of business services - a business service may invoke other platform/utility services to perform database read and writes, to perform service authorization etc.

We have deployed these services as WAR files in Tomcat.

We want to test this whole setup using a integration test suite which we would like to also treat as regression test suite.

What would be a good approach to perform integration testing on this and any tools that can speed up the development of suite? Here are few requirements we think we need to address:

1. Ability to define integration test cases which exercise business scenarios.
2. Set up the DB with test data before suite is run.
3. Invoke the REST API that is running on a remote server (Tomcat)
4. Validate the DB post test execution for verifying expected output
5. Have code coverage report of REST API so that we know how confident we should be in the scenarios covered by the suite.

回答1:

At my work we have recently put together a couple of test suites to test some RESTful APIs we built. Like your services, ours can invoke other RESTful APIs they depend on. We split it into two suites.

---

• Suite 1 - Testing each service in isolation
  • Mocks any peer services the API depends on using restito. Other alternatives include rest-driver, wiremock, pre-canned and betamax.
  • The tests, the service we are testing and the mocks all run in a single JVM
  • Launches the service we are testing in Jetty

I would definitely recommend doing this. It has worked really well for us. The main advantages are:

• Peer services are mocked, so you needn't perform any complicated data setup. Before each test you simply use restito to define how you want peer services to behave, just like you would with classes in unit tests with Mockito.
• The suite is super fast as mocked services serve pre-canned in-memory responses. So we can get good coverage without the suite taking an age to run.
• The suite is reliable and repeatable as its isolated in it's own JVM, so no need to worry about other suites/people mucking about with an shared environment at the same time the suite is running and causing tests to fail.

---

• Suite 2 - Full End to End
  • Suite runs against a full environment deployed across several machines
  • API deployed on Tomcat in environment
  • Peer services are real 'as live' full deployments

This suite requires us to do data set up in peer services which means tests generally take more time to write. As much as possible we use REST clients to do data set up in peer services.

Tests in this suite usually take longer to write, so we put most of our coverage in Suite 1. That being said there is still clear value in this suite as our mocks in Suite 1 may not be behaving quite like the real services.

---

With regards to your points, here is what we do:

• Ability to define integration test cases which exercise business scenarios.
  • We use cucumber-jvm to define business scenarios for both of the above suites. These scenarios are English plain text files that business users can understand and also drive the tests.
• Set up the DB with test data before suite is run.
  • We don't do this for our integration suites, but in the past I have used unitils with dbunit for unit tests and it worked pretty well.
• Invoke the REST API that is running on a remote server (Tomcat)
  • We use rest-assured, which is a great HTTP client geared specifically for testing REST APIs.
• Validate the DB post test execution for verifying expected output
  • I can't provide any recommendations here as we don't use any libraries to help make this easier, we just do it manually. Let me know if you find anything.
• Have code coverage report of REST API so that we know how confident we should be in the scenarios covered by the suite.
  • We do not measure code coverage for our integration tests, only for our unit tests, so again I can't provide any recommendations here.

Keep your eyes peeled on our techblog as there may be more details on their in the future.

回答2:

You may also check out the tool named Arquillian, it's a bit difficult to set up at first, but provides the complete runtime for integration tests (i.e. starts its own container instance and deploys your application along with the tests) and provides extensions that solve your problems (invoking REST endpoints, feeding the databases, comparing results after the tests).

Jacoco extension generates the coverage reports than can be later displayed i.e. by the Sonar tool.

I've used it for a relatively small-scale JEE6 project and, once I had managed to set it up, I was quite happy with how it works.