Integration test per layer is a good practice?

问题:

I have an application that use spring-mvc, basically we have a presentation layer (controllers), service layer (business units, helpers), integration layer and data access layer(jdbc/jpa repositories), we want to ensure using testing that future addition to the code won't break nothing that was previously working, to do this we are using unit testing(mockito) and integration testing (spring-test,spring-test-mvc).

Unit testing is made per class/component, basically we tried to have a good coverage for the incoming inputs and possible flows within these components and this action is working fine, not have doubts here as unit test is about ensure the units works as expected.

Integration test is different story and very debatable one, as for now we are using sometimes the same scenarios we use to design our unit testing but having the entire system available using real platform and so on, but I have doubts about the best practices here.

1. As we have a controller, service, data layer one approach is made an IT per layer, example we have UserService class, we will have UserServiceTest which will be the Unit test and UserServiceIT, but maintainability is not ideal, I feel sometimes we repeat the same test scenario but now using the real system. Does this practice really make sense or in which scenarios this makes sense ?. If we already have 100% test coverage in the class with unit testing why we need IT for this one, seems that we have this only to ensure real component is going to start-up ?, Make sense to have all the same scenarios or which is a good criteria to decide?

2. Other approach is just go with the most important test cases via integration test but just from the controller layer, which it means invoke the REST services and verify the JSON output. This is enough ?, we don't need to verify more things in the others layers ?. I know calling the real REST api will use underneath all the layers (controller, service, dao) but this is enough ? Some consideration you will say here?

3. If we have a helper class I don't think make sense to have unit and IT, as most of the method as there for only one purpose I think unit testing will be enough here, does you think the same?.

4. Some classes in the data layer could use Criteria API, QueryDSL for those I go using IT as make the unit testing in some cases is extremely difficult, this is a valid justification?

I am trying to get the best way, tips and practices that makes the task of ensure the system integrity a real and valuable process keeping in mind the maintainability of this stuff.

回答1:

You kindda touch the entire Test strategy needed for your application. Testing is not only about coverage and layers. As example:

we want to ensure using testing that future addition to the code won't break nothing that was previously working, to do this we are using unit testing(mockito) and integration testing (spring-test,spring-test-mvc).

this is how you actually support Regression testing, which is a type. If we look at the (detailed) Test pyramid

it's easy to see that the integration tests take good portion (recommended 5-15%). Integration goes cross-layer, but also cross-componentAPIs. It's natural that your business components will live in same layer, but you still need to assure that they are working as expected with each other too. Having mSOA will push you to support such extensive interfaces integration testing.

I agree with you on this one

Integration test is different story and very debatable

Some experts even suggest that you have to keep only unit tests and the GUI E2E ones. IMHO there are no strict best practices - only good ones. If you are happy with the trade-offs, use what ever suits your case.

I feel sometimes we repeat the same test scenario but now using the real system. Does this practice really make sense or in which scenarios this makes sense ? If we already have 100% test coverage in the class with unit testing why we need IT for this one, seems that we have this only to ensure real component is going to start-up ? Make sense to have all the same scenarios or which is a good criteria to decide?

It looks like you need to draw a line in those scenarios. Keeping the long story short - unit testing and Mock objects go together naturally. Component tests will require some real system behavior, it can be used to check the handling of data passed between various units, or subsystem components - like your component/service DB or messaging that is not an unit level task.

from the controller layer, which it means invoke the REST services and verify the JSON output. This is enough ?, We don't need to verify more things in the others layers ?. I know calling the real REST api will use underneath all the layers (controller, service, dao) but this is enough ?

Not quite true - testing the presentation layer will exercise the underlying layers too ... so why bother with all the rest of the testing? If you are OK with such approach - Selenium team suggests such DB validation approach.

If you're talking about Beans and ViewHelpers here

we have a helper class I don't think make sense to have unit and IT, as most of the method as there for only one purpose I think unit testing will be enough here, does you think the same?.

you'll need both unit and IT, because all the reasons valid for other components. Having Single responsibility doesn't deny need of IT testing.

make the unit testing in some cases is extremely difficult, this is a valid justification?

Same goes for all your encapsulated private (and static) classes, methods, properties etc. But there is a way of testing those as well - like reflection. This of course is for a special case of unit testing legacy code or an API you can't change. If you needed it for your own code, maybe this lack of testability points to a design smell.

回答2:

The approach I would recommend, based on recent experience of testing Java EE 7 and Spring-based codebases, is:

Use per-feature integration tests, and avoid unit tests and mocking. Each integration test should cover code from all layers, from the presentation layer down to the infrastructure layer (this one containing reusable components that are not application or domain specific, but appropriate to the chosen architecture).

Most integration tests should be based on actual business requirements and input data. Others may be created to exercise remaining parts of the codebase, according to the code coverage report generated from each execution of the integration test suite.

So, assuming "full" code coverage is achieved with integration tests, and they run sufficiently fast, there isn't much reason to have unit tests at all. My experience is that when writing unit tests, developers tend to use too much mocking, often creating brittle tests that verify unnecessary implementation details. Also, unit tests can never provide the same level of confidence as integration tests can, since they usually don't cover things like database queries, ORM mapping, and so on.

回答3:

Unit testing applies as you did on classes and components. Its purpose is to:

• Write code (TDD).
• Illustrate the code usage and make it sustainable over time and changes.
• Cover as much border cases as possible.

When you encounter an issue with some specific usage or parameters, first reproduce it with a new test, then fix it.

Mocking should only be used when it is needed to test a class or component standalone behavior and the mocked feature comes in production from outside your application (an email server for instance). It is overkill and useless when the code is already covered and the mocking overlaps the responsibility of other kind of tests, such as integration tests.

Now that you know every piece of code works, how do the pieces work together? This is where comes the integration testing which is about how the components interact together in various conditions and environments. There is sometimes little difference between UT and IT: for instance the testing of the data access layer. Integration tests are used for the same purposes as unit testing but at a higher level, less atomic, to illustrate the use cases on APIs, services...

What do you call the "integration layer"?

The presentation layer testing is rather the responsibility of functional testing, not unit nor integration.

You also did not talk about performance testing.

Finally, the goal is getting all code wrote along with the tests, bugs fixed after reproduction with new tests, and maximum coverage cumulating the all kinds of tests in all possible conditions (OS, databases, browsers...). So you validate your overall testing quality with:

• a tool calculating the coverage. You will likely have to instrumentate the code to evaluate the coverage from functional testing or use advanced JDK tools.
• the number of bugs coming from lack of tests on some components, services...

I usually consider a bunch of tests being good when reading them gives me immediately no doubt about how to use the code they cover and full confidence to its contract, capabilities, inputs and outputs, history and exhaustibility on use cases, strength and stability in regard to error management and report.

Not also the coverage is one important thing, but it could be better to have a few less tests if you focus on their quality: threadsafe, made of unordered methods and classes, testing real conditions (no "if test condition" hacks).

To answer your question: I would say that given the above considerations, you don't have to write an integration test per layer since you will rather choose a different testing strategy (unit, integration, functional, performance, smoke, mocked...) for each layer.