Best practices for the best max length of time for

Question

We are doing continuous integration at our company with TeamCity and we have unit tests running at every commit (1 min window).

Lately, we are debating on how long a batch of unit tests should last but the shortest the better.

However, I would like to know what is the best practice for the length of a batch of unit tests?

Answer 1

You could build priorities into your unit tests, and only use a subset as a check-in gate (Build Verification Test, or BVT). Run lower priority tests less often (e.g. per daily build, per test pass, or per product release). Then place separate execution time limits on each (or each suite) that satisfies your dev team.

I base priorities on how fast we'd jump on fixing the bug signaled by a test failure. P0 means "must-fix, even if we have to slip the schedule", P3 means "may never fix".

One of the teams I worked on said no more than 2 minutes per feature for BVTs, and placed no time restriction on lower priority tests. The devs had to run about 5 test suites, and it was reasonable with our volume of check-ins to queue up 10 minute buddy builds. But our "unit tests" were big-huge, special-environment-required integration tests, so YMMV.

Answer 2

Very simple make the (unit-)tests faster or parallelize them...unit tests should work...You shouldn't limit them by runtime...

Answer 3

Unit tests should be run until they are all complete; don't restrict the set of unit tests based upon the runtime. If you have lots of unit tests, and they're taking a long time to run, investigate getting faster hardware to run the CI system on; buying more expensive hardware is far cheaper than not putting in place the unit tests that would detect a problem before it becomes a major bug.

Answer 4

"As short as possible."

But really, it depends on exactly what you're asking. Should you remove tests to shorten the build? Probably not. Might you limit the scope of the tests run on a per-commit build? Maybe so. Should you limit the tests run on a nightly build? Probably not. Exactly how long it's okay for a build to take really depends on your team, your process, and how you integrate CI into them.