Code coverage coterie confab

Every once in a while the topic of code coverage surfaces, which invariably leads to a number of interesting views and comments. Recently, my friend Meera Subbarao mused about the subject by rhetorically questioning “Is Code Coverage Important?”

As Meera points out, code coverage unveils a number of interesting aspects about code including:

how effective our tests are, what parts of our source code are thoroughly executed.

Yet, the most important facet of code coverage is that which isn’t measured; that is:

You can also look at the code coverage report and find out specific areas of code which are not exercised by our tests.

And therein lies the most telling metric that a code coverage report can covey — that which isn’t tested.

Interesting, Meera’s article generated quite a lot of comments, which are all excellent reads as various members of the code coverage coterie weigh in, including Rasmus Grouleff, who notes:

You can achieve 100% coverage without writing a single assert, so coverage as a measure of code quality isn’t really all that good, unless you factor in some sort of measure of how good the tests are along with different kinds of metrics such as cyclomatic complexity, lines of code per method and so on.

Which, of course, is exactly why the numbers code coverage by itself provides are essentially useless– the real value is in what a code coverage report doesn’t divulge– that which isn’t tested!

Interesting, npellow asserts that:

I would rather have a project with low code coverage, and high quality tests, than one with high coverage and low quality tests. Since coverage is a negative metric, the uncovered code in a coverage report actually holds more information than the covered lines. ie if a statement is uncovered, you can be certain it is not tested. The same is not true for a covered line.

Indeed, and even more dangerous is the fact that a hip conditional, while actually “touched” still isn’t proven to be correct– especially if that conditional has a short-circuit operation in it!

Jean-Francois adroitly pointed out that:

having specific unit tests providing 100% coverage of Java Beans methods is generally a total waste of time. Whereas complex methods (with business rules) should call for extra testing efforts.

Right on, baby! Hence, metrics alone are usually boring– it is the combination of various metrics that imparts real meaning.

And weighing in on the value of percent coverage numbers, my good friend Alex Ruiz remarks:

I use code coverage tools as a reference only. I don’t have a specific percentage in mind. What I do is look at the least tested code, and depending on its complexity or how critical that code is, I decide to add a test or not. In another words, I don’t aim for a specific number blindly. I use coverage only to analyze where tests are needed.

You got it, man! Indeed, the confab in this article alone is a wealth of great information regarding code coverage. Once you’ve pondered this article and its manifold comments, take a peek at my friend Jason Rudolph’s blog write-up entitled “Testing Anti-Patterns Potpourri – Quotes, Resources, and Collective Wisdom” — this is a veritable Holy Writ on all things related to where we stand as an industry on testing and related metrics.

The subject of code coverage pops up from time to time– as to whether or not it is important is a matter of opinion; suffice to say, what’s most important is what a coverage report doesn’t tell you directly. As I wrote about in IBM developerWorks’ “Don’t be fooled by the coverage report“, the value of a code coverage report is in exposing

code that hasn’t been tested, on a micro level and on a macro level. You can facilitate deeper coverage testing by analyzing your code base from the top level as well as analyzing individual class coverage. Once you’ve integrated this principle you and your organization can use coverage measurement tools where they really count, such as to estimate the time needed for a project, continuously monitor code quality, and facilitate QA collaboration.

Can you dig it, man?