What you need to know about test coverage

Recently, I posted a blog about the ROI of test automation and its benefits. I mentioned that it increases test coverage, but do we really know what is test coverage?

In this post, I’ll go into more detail about it.

Basically, it’s one of the measurements of test quality that tells us how much code has been tested. It defines certain entities of the system with the intent to cover them with tests. It’s a way to indicate when we have tested sufficiently and gives us ideas of what else to test (thus expanding coverage).

Consider carrying out tests to be like sweeping the floors of your house. I always forget to sweep some part, like the upstairs bathroom, so my coverage does not include that room. Imagine if I only included sweeping bedrooms for my sweep coverage criteria.

With that criteria, if I sweep 100% of the bedrooms, does that mean that the whole house is clean?

No, because I completely missed the kitchen, dining room, bathrooms, etc! Therefore, be careful and measure test coverage with caution. To have a certain level of coverage is an indicator of the quality of the tests, but it is never an indicator of the quality of the system, neither does it guarantee that everything has been tested. Test coverage tells us what percentage of the code has been tested, but it doesn’t mean that it has been tested in every situation either.

What is Test Coverage Good For?

In this case, test coverage is good for:

• Measuring the quality of sweeping
• Indicating when to stop sweeping
• Alerting me of other places that need to be swept

Certain criteria can be more powerful than other criteria. Knowing them can give you an indicator of how deep the tests are and when to apply one criterion or another. For example, it is said that a criteria A includes another criteria B if any set of test cases TS that covers criteria A also covers criteria B.

For sweeping, we may want to follow the following criteria:

1. Sweep every bedroom
2. Sweep every part of the house (bedroom, bathroom, kitchen, etc)
3. Sweep every tiny spot, even the corners, because they are likely to accumulate dirt.

As you can see, Criterion 3 includes 2, which includes 1 (the relationship is transitive as number 3 includes number 1. If we design a test case for criterion number 3, it should also cover the first two criteria. For testing software, the criteria usually include the various paths, conditions, statements, functions, etc. within a program.

Another real example could be, for instance, equivalence class partitioning, where classes are defined and then one element from each class is selected, and in this way, we cover all the classes. If we think about white-box testing, we have sentence coverage, branch coverage, path coverage, etc., and in particular, for state machines, we have criteria that indicate to cover all nodes, all transitions, etc.

Where Does Test Automation Fit Into All This?

Well, imagine you ditched the broom and replaced it with a super speedy robot. You would manage to clean the floors faster and miss fewer spots, also allowing you to focus on more important things while it does the work for you.

Do you measure test coverage?

Can you imagine how test automation would help to improve this measurement? Get started with test automation today!

Recommended for You

Webinar Summary: How to Use Automation to Increase Test Coverage
The True ROI of Test Automation