Are C++ static code analyis tools worth it?

Question

Our management has recently been talking to some people selling C++ static analysis tools. Of course the sales people say they will find tons of bugs, but I'm skeptical.

How do such tools work in the real world? Do they find real bugs? Do they help more junior programmers learn?

Are they worth the trouble?

Answer 1

At a former employer we had Insure++. It helped to pinpoint random behaviour (use of uninitialized stuff) which Valgrind could not find. But most important: it helpd to remove mistakes which were not known as errors yet.

Insure++ is good, but pricey, that's why we bought one user license only.

Answer 2

As with everything the answer depends ... if you are the sole developer working on a knitting-pattern-pretty-printer for you grandma you'll probably do not want to buy any static analysis tools. If you are having a medium sized project for software that will go into something important and maybe on top of that you have a tight schedule, you might want to invest a little bit now that saves you much more later on.

I recently wrote a general rant on this: http://www.redlizards.com/blog/?p=29

I should write part 2 as soon as time permits, but in general do some rough calculations whether it is worth it for you:

• how much time spent on debugging?
• how many resources bound?
• what percentage could have been found by static analysis?
• costs for tool setup?
• purchase price?
• peace of mind? :-)

My personal take is also:

• get static analysis in early

  • early in the project
  • early in the development cycle
  • early as in really early (before nightly build and subsequent testing)

• provide the developer with the ability to use static analysis himself

  • nobody likes to be told by test engineers or some anonymous tool what they did wrong yesterday
  • less debugging makes a developer happy :-)
  • provides a good way of learning about (subtle) pitfalls without embarrassment

Answer 3

Those tools do help. lint has been a great tool for C developers.

But one objection that I have is that they're batch processes that run after you've written a fair amount of code and potentially generate a lot of messages.

I think a better approach is to build such a thing into your IDE and have it point out the problem while you're writing it so you can correct it right away. Don't let those problems get into the code base in the first place.

That's the difference between the FindBugs static analysis tool for Java and IntelliJ's Inspector. I greatly prefer the latter.

Answer 4

Paying for most static analysis tools is probably unnecessary when there's some very good-quality free ones (unless you need some very special or specific feature provided by a commercial version). For example, see this answer I gave on another question about cppcheck.

Answer 5

In my experience with a couple of employers, Coverity Prevent for C/C++ was decidedly worth it, finding some bugs even in good developers’ code, and a lot of bugs in the worst developers’ code. Others have already covered technical aspects, so I’ll focus on the political difficulties.

First, the developers whose code need static analysis the most, are the least likely to use it voluntarily. So I’m afraid you’ll need strong management backing, in practice as well as in theory; otherwise it might end up as just a checklist item, to produce impressive metrics without actually getting bugs fixed. Any static analysis tool is going to produce false positives; you’re probably going to need to dedicate somebody to minimizing the annoyance from them, e.g., by triaging defects, prioritizing the checkers, and tweaking the settings. (A commercial tool should be extremely good at never showing a false positive more than once; that alone may be worth the price.) Even the genuine defects are likely to generate annoyance; my advice on this is not to worry about, e.g., check-in comments grumbling that obviously destructive bugs are “minor.”

My biggest piece of advice is a corollary to my first law, above: Take the cheap shots first, and look at the painfully obvious bugs from your worst developers. Some of these might even have been found by compiler warnings, but a lot of bugs can slip through those cracks, e.g., when they’re suppressed by command-line options. Really blatant bugs can be politically useful, e.g., with a Top Ten List of the funniest defects, which can concentrate minds wonderfully, if used carefully.

Answer 6

It does help. I'd suggest taking a trial version and running it through a part of your codebase which you think is neglected. These tools generate a lot of false positives. Once you've waded through these, you're likely to find a buffer overrun or two that can save a lot of grief in near future. Also, try at least two/three varieties (and also some of the OpenSource stuff).