What is a reasonable code coverage % for unit tests (and why)?

This prose by Alberto Savoia answers precisely that question (in a nicely entertaining manner at that! ).

This prose by Alberto Savoia answers precisely that question (in a nicely entertaining manner at that! ): artima.com/forums/flat.jsp?forum=106&thr... Testivus On Test Coverage Early one morning, a programmer asked the great master: “I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied: “Don’t worry about coverage, just write some good tests. ” The programmer smiled, bowed, and left. ... Later that day, a second programmer asked the same question.

The great master pointed at a pot “How many grains of rice should I put in that pot? ” The programmer, looking puzzled, replied: “How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.” “Exactly,” said the great master.

The second programmer smiled, bowed, and left. ... Toward the end of the day, a third programmer came and asked the same question about code coverage. “Eighty percent and no less!

” Replied the master in a stern voice, pounding his fist on the table. The third programmer smiled, bowed, and left.... After this last reply, a young apprentice approached the great master: “Great master, today I overheard you answer the same question about code coverage with three different answers. Why?

” The great master stood up from his chair: “Come get some fresh tea with me and let’s talk about it.” After they filled their cups with smoking hot green tea, the great master began to answer: “The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests.

He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.

” “The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.

” “I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?” The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down. “The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.

” The young apprentice and the grizzled great master finished drinking their tea in contemplative silence. Update The point of this whole anecdote is that you should try and not focus on the coverage percentage per se, or try to find an arbitrary number for it, but instead focus on having as much logic and functionality tested as is humanly possible. It is quite reasonable to have a, say, 50% coverage rate if only because only 50% of the code contains logic that can be tested, and the other 50% happens to be simple DTOs or things that are handled by a framework (you don't need to test the functionalities of your framework).

2 Sounds like an argument against the general concept of code coverage, as a metric for evaluating the usefulness of unit tests. I'm sure everyone agrees it isn't a perfect metric, but personal experience should hopefully show some correlation between CC % and unit test effectiveness... – sanity Sep 18 '08 at 4:34 sanity -- your statement is mirrored precisely by the response to the "second developer". Personal experience should dictate it.

– Jon Limjap Sep 18 '08 at 4:36 I see, so you think that there is no valuable information that can be shared other than "make a decision based on your personal experience"? – sanity Sep 18 '08 at 4:46 2 No, but assigning an arbitrary percentage to a problem that is quite, should I say, malleable, would not be prudent. The "reasonable" part of the question appears to be subjective.

– Jon Limjap Sep 18 '08 at 5:05 7 Perfect answer. Metrics do not make good code. You can write crappy code with 100% coverage and it doesn't make the code work good.

+1 from me, shame I can't up more :) – Rob Cooper Sep 18 '087 at 10:14.

Code Coverage is a misleading metric if 100% coverage is your goal (instead of 100% testing of all features). You could get a 100% by hitting all the lines once. However you could still miss out testing a particular sequence (logical path) in which those lines are hit.

You could not get a 100% but still have tested all your 80%/freq used code-paths. Having tests that test every 'throw ExceptionTypeX' or similar defensive programming guard you've put in is a 'nice to have' not a 'must have' So trust yourself or your developers to be thorough and cover every path through their code. Be pragmatic and don't chase the magical 100% coverage.

If you TDD your code you should get a 90%+ coverage as a bonus. Use code-coverage to highlight chunks of code you have missed (shouldn't happen if you TDD though.. since you write code only to make a test pass. No code can exist without its partner test.).

– MrWiggles Jan 30 '09 at 12:17 1 Exceptions should be exceptional - not supposed to happen. If they do, you log the point of failure and bail. You can't test every exception that could happen.

If the app is supposed to handle a non-happy path/event, you should have a test for it. Accessors may be added for future clients.. depends – Gishu Jan 30 '09 at 13:40 I take back my words: Exceptions need not be exceptional - a method may throw an exception to indicate that it could not carry out its work to completion. What I meant was you don't need a test for every method where you've put a defensive just-in-case try-catch.

All known failure modes of a method must be tested with automated UTs – Gishu Jun 26 '09 at 7:21 I'm not sure what you mean by your second point "but still have tested all your code-paths". If you in fact mean full-path coverage, then no you cannot have full-path coverage without 100% line/branch/decision coverage. In fact, full-path coverage is usually unobtainable in any non-trivial program because of the combinatoric nature of branches in generating paths.

En.wikipedia.org/wiki/Code_coverage#Other_coverage_criteria – Zach Burlingame Apr 11 '11 at 23:40 @Zach - my mistake. I should have said "all code paths that you want to work" or "non-fringe paths". If the path failure is a very rare possibility I would be okay with it not being covered.

– Gishu Apr 15 '11 at 7:04.

85% would be a good starting place for checkin criteria. I'd probably chose a variety of higher bars for shipping criteria - depending on the criticality of the subsystems/components being tested.

– sanity Sep 18 '08 at 4:29 As a footnote - this can be messy for projects where automation is difficult - as always be pragmatic about what is achievable vs. desireable. – stephbu Sep 18 '08 at 4:29 1 Mainly through experimentation.It is pretty easy to get to code coverage to 80-90% for Dev-related unit tests - going higher normally needs divine test intervention - or really simple code paths. – stephbu Sep 18 '08 at 4:31.

I'd have another anectode on test coverage I'd like to share. We have a huge project wherein, over twitter, I noted that, with 700 unit tests, we only have 20% code coverage. Scott Hanselman replied with words of wisdom: Is it the RIGHT 20%?

Is it the 20% that represents the code your users hit the most? You might add 50 more tests and only add 2%. Again, it goes back to my Testivus on Code Coverage Answer.

How much rice should you put in the pot? It depends.

Obviously there has to be common sense in there. Its not much use if the 50% of the code you are testing are comments. – sanity Sep 18 '08 at 4:48 It's more in the lines of... is your coverage spent on your application's core functionality, or is it uselessly testing trivial features/nice-to-haves?

– Jon Limjap Sep 18 '08 at 5:07.

Check out Crap4j. It's a slightly more sophisticated approach than straight code coverage. It combines code coverage measurements with complexity measurements, and then shows you what complex code isn't currently tested.

It depends greatly on your application. For example, some applications consist mostly of GUI code that cannot be unit tested.

1 You should probably use Model View Presenter for you UI if you are in a TDD enviernment. – Charles Graham Sep 18 '08 at 4:30.

I don't think there can be such a B/W rule. Code should be reviewed, with particular attention to the critical details. However, if it hasn't been tested, it has a bug!

Don't want a rule, just feedback on any personal experience on the correlation between code-coverage percentage and unit test effectiveness. – sanity Sep 18 '08 at 4:35.

Depending on the criticality of the code, anywhere from 75%-85% is a good rule of thumb. Shipping code should definitely be tested more thoroughly than in house utilities, etc.

I think the best symptom of correct code coverage is that amount of concrete problems unit tests help to fix is reasonably corresponds to size of unit tests code you created.

If you've been doing unit testing for a decent amount of time, I see no reason for it not to be approaching 95%+. However, at a minimum, I've always worked with 80%, even when new to testing. This number should only include code written in the project (excludes frameworks, plugins, etc.) and maybe even exclude certain classes composed entirely of code written of calls to outside code.

This sort of call should be mocked/stubbed.

My answer to this conundrum is to have 100% line coverage of the code you can test and 0% line coverage of the code you can't test. My current practice in Python is to divide my . Py modules into two folders: app1/ and app2/ and when running unit tests calculate the coverage of those two folders and visually check (I must automate this someday) that app1 has 100% coverage and app2 has 0% coverage.

When/if I find that these numbers differ from standard I investigage and alter the design of the code so that coverage conforms to the standard. This does mean that I can recommend achieving 100% line coverage of library code. I also occasionally review app2/ to see if I could possible test any code there, and If I can I move it into app1/ Now I'm not too worried about the aggregate coverage because that can vary wildly depending on the size of the project, but generally I've seen 70% to over 90%.

With python, I should be able to devise a smoke test which could automatically run my app while measuring coverage and hopefully gain an aggreagate of 100% when combining the smoke test with unittest figures.

Code coverage is great but only as long as the benefits that you get from it outweigh the cost/effort of achieving it. We have been working to a standard of 80% for some time, however we have just made the decison to abandon this and instead be more focused on our testing. Concentrating on the complex business logic etc, This decision was taken due to the increasing amount of time we spent chasing code coverage and maintaining existing unit tests.

We felt we had got to the point where the benefit we were getting from our code coverage was deemed to be less than the effort that we had to put in to achieve it.

I use cobertura, and whatever the percentage, I would recommend keeping the values in the cobertura-check task up-to-date. At the minimum, keep raising totallinerate and totalbranchrate to just below your current coverage, but never lower those values. Also tie in the Ant build failure property to this task.

If the build fails because of lack of coverage, you know someone's added code but hasn't tested it. Example.

I would agree, anything above 80-85% is pretty good.

If this were a perfect world, 100% of code would be covered by unit tests. However, since this is NOT a perfect world, it's a matter of what you have time for. As a result, I recommend focusing less on a specific percentage, and focusing more on the critical areas.

If your code is well-written (or at least a reasonable facsimile thereof) there should be several key points where APIs are exposed to other code. Focus your testing efforts on these APIs. Make sure that the APIs are 1) well documented and 2) have test cases written that match the documentation.

If the expected results don't match up with the docs, then you have a bug in either your code, documentation, or test cases. All of which are good to vet out. Good luck!

Short answer: 60-80% Long answer: I think it totally depends on the nature of your project. I typically start a project by unit testing every practical piece. By the first "release" of the project you should have a pretty good base percentage based on the type of programming you are doing.At that point you can start "enforcing" a minimum code coverage.

We were targeting >80% till few days back, But after we used a lot of Generated code, We do not care for %age, but rather make reviewer take a call on the coverage required.

This has to be dependent on what phase of your application development lifecycle you are in. If you've been at development for a while and have a lot of implemented code already and are just now realizing that you need to think about code coverage then you have to check your current coverage (if it exists) and then use that baseline to set milestones each sprint (or an average rise over a period of sprints), which means taking on code debt while continuing to deliver end user value (at least in my experience the end user doesn't care one bit if you've increased test coverage if they don't see new features). Depending on your domain it's not unreasonable to shoot for 95%, but I'd have to say on average your going to be looking at an average case of 85% to 90%.

Generally speaking, from the several engineering excellence best practices papers that I have read, 80% for new code in unit tests is the point that yields the best return. Going above that CC% yields a lower amount of defects for the amount of effort exerted. This is a best practice that is used by many major corporations.

Unfortunately, most of these results are internal to companies, so there are no public literatures that I can point you to.

Viewing coverage from another perspective: Well-written code with a clear flow of control is the easiest to cover, the easiest to read, and usually the least buggy code. By writing code with clearness and coverability in mind, and by writing the unit tests in parallel with the code, you get the best results IMHO.

I think that what may matter most is knowing what the coverage trend is over time and understanding the reasons for changes in the trend. Whether you view the changes in the trend as good or bad will depend upon your analysis of the reason.

Code coverage is great, but functionality coverage is even better. I don't believe in covering every single line I write. But I do believe in writing 100% test coverage of all the functionality I want to provide (even for the extra cool features I came with myself and which were not discussed during the meetings).

I don't care if I would have code which is not covered in tests, but I would care if I would refactor my code and end up having a different behaviour. Therefore, 100% functionality coverage is my only target.

When I think my code isn't unit tested enough, and I'm not sure what to test next, I use coverage to help me decide what to test next. If I increase coverage in a unit test - I know this unit test worth something. This goes for code that is not covered, 50% covered or 97% covered.

In my opinion, the answer is "It depends on how much time you have". I try to achieve 100% but I don't make a fuss if I don't get it with the time I have. When I write unit tests, I wear a different hat compared to the hat I wear when developing production code.

I think about what the tested code claims to do and what are the situations that can possible break it. I usually follow the following criteria or rules: That the Unit Test should be a form of documentation on what's the expected behavior of my codes, ie. The expected output given a certain input and the exceptions it may throw that clients may want to catch (What the users of my code should know?) That the Unit Test should help me discover the what if conditions that I may not yet have thought of.

(How to make my code stable and robust? ) If these two rules doesn't produce 100% coverage then so be it. But once, I have the time, I analyze the uncovered blocks and lines and determine if there are still test cases without unit tests or if the code needs to be refactored to eliminate the unecessary codes.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.