Only run unit tests which's respective source

问题:

I am running unit tests and Selenium tests in our Jenkins CI server. As we all know, tests take long to run in a large project.

Is there a tool/framework for Java which could only trigger tests whose respective source code has changed? This because not every commit to SCM affects all areas of the source code...

I am using Cobertura for code coverage and Surefire for reporting.

EDIT: I found Atlassian Clover, but I am searching for a free solution.

回答1:

I am running unit tests and Selenium tests in our Jenkins CI server. As we all know, tests take long to run in a large project.

This is the problem I'd tackle. Split your project into multiple logical units (e.g. Persistence Layer, Service Layer, Web Layer) and test them individually. That way you only need to run Selenium tests when the web layer has changed and the build time per artifact grows shorter

回答2:

You could try JUnit Max or Infinitest, but they're both IDE-based.

回答3:

In general you cannot reliably infer what code is covered by what test without running it, and even when you do run your tests the execution path could be different from run to run. I suspect perfect analysis like this is equivalent to Halting Problem.

However you can infer what tests are likely to exercise changed code using either coverage from previous test run or imports in the test case files.

In the end, you do want to run your entire test suite, however, you are right that running tests over changed code is more important because 1. developer knows they messed up earlier and 2. if you experience a lot of changes, it makes sense to test newest revision first and go to earlier revisions only to bisect problems.

Google famously claimed * that their hacked code repository runs unit tests at commit time ** and that they tweaked the test framework to determine what tests to run first. I don't know if their changes are public.

* video talk to conference presentation, I don't recall exact name or link

** strictly after commit, followed by automatic rollback if tests begin to fail

回答4:

Personally I think you're looking at this the wrong way. What would be better to look at would be breaking down your Junit tests into separate packages based off of functionality and time.

For projects like this I run sanity tests for each commit, but these tests are very limited and cover the most basic of functions and then on top of this run nightly tests to cover much more (seeing as no-one works nights, it gives a 12 hour window to test that days code).

If your entire testing suite required it (but I would be surprised if it did) you could then run a weekend test suite.

To test only altered classes would assume that these classes don't affect any other parts of your project, which could lead to errors that wouldn't be picked up unless you did comprehensive tests, or you just happened to be altering the affected area anyway.

I know this doesn't technically answer your question, but something to think over.