{"id":6343,"date":"2016-05-23T17:10:05","date_gmt":"2016-05-23T17:10:05","guid":{"rendered":"http:\/\/www.abstracta.us\/?p=6343"},"modified":"2025-05-05T21:21:16","modified_gmt":"2025-05-05T21:21:16","slug":"optimize-test-coverage-long-term","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/","title":{"rendered":"How to Optimize Test Coverage in the Long-Term"},"content":{"rendered":"<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n<h1><span style=\"font-weight: 400; color: #333333;\">Our method for improving test coverage over multiple test cycles<\/span><\/h1>\n<p><span style=\"font-weight: 400; color: #333333;\">We want to test as much code as humanly (or mechanically) possible right? Yes and no. For each test cycle, it\u2019s important to consider multiple strategies for measuring test coverage and to put a system into place where it can be maximized in the long-term as well.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Test coverage is one of the measurements of test quality that tells us how much of the application under test has been tested. As I explained in a past <a href=\"http:\/\/abstracta.us\/blog\/test-automation\/they-say-automation-increases-test-coverage-but-what-is-test-coverage\/\">post<\/a>, it&#8217;s a little bit like sweeping the floors of a home. When I sweep my house, I always forget to sweep some part, like the upstairs bathroom, so my sweep coverage fails to include that room. Imagine if I only included sweeping bedrooms for my sweep coverage criteria. With that criteria, if I swept 100% of the bedrooms, would that mean that the whole house is clean? No, because I completely missed the kitchen, dining room, bathrooms, etc! Therefore, we must always be careful with test coverage and recognize that sometimes it has its limitations.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">We find test coverage useful for defining certain entities of the system with the intent to cover them with tests. It also tells us when we have tested sufficiently, gives us ideas of what else to test (thus expanding coverage), and helps us know quantitatively the extent of our tests. It\u2019s a great measuring stick, but <\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">[tweet_box design=&#8221;default&#8221; float=&#8221;none&#8221;]Even with 100% coverage we will never be guaranteed that our application is bug-free.[\/tweet_box]<\/span><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">It\u2019s also important to note that the percentage of test coverage isn\u2019t everything. Even if you only managed to achieve 20% coverage, it wouldn\u2019t be a bad thing because the ideal amount of test coverage should be determined after running a risk analysis and should be in line with your priorities.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">There are many ways to consider test coverage. Here I am going to look at code coverage, data-oriented coverage, and the plethora of other techniques at a tester\u2019s disposal.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Code_Coverage\"><\/span>Code Coverage<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">Code coverage is the most popular metric for measuring coverage. There are different levels- not only lines of code covered, but there are also branches, decisions inside logic constructors, etc.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Data-Oriented_Coverage\"><\/span>Data-Oriented Coverage<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">With data-oriented coverage, you have input and output parameters, each of them having their own domain (the spectrum of possible values they can have) and if you think about all the possibilities, you see you end up with a cartesian product (because you can test every possible combination). On the other hand, you can test less and go with \u201ceach choice\u201d coverage, which means that you choose at least each possible value at least once. There is also all-pairs, which is empirically said to have the best cost-benefit relationship, being the best mix between each-choice and all combinations.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Other_Kinds_of_Coverage\"><\/span>Other Kinds of Coverage<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">In addition to those previously mentioned, there are several more ways to cover the product that you are testing such as state-machines, decision tables, decision trees, equivalence partition, and boundary values, etc. It\u2019s very interesting to see that each technique is supported by an \u201cerror theory\u201d. The error theory takes into account the typical errors that programmers commit, as for example, equivalence partition and boundary values consider the error of, for example, using a \u201c&lt;\u201d instead of a \u201c&lt;=\u201d, misunderstanding business logic, or for example, decision tree tries to execute all the \u201cdecisions\u201d and combinations of interesting conditions that a program has, which is nothing more than trying to maximize the branch coverage in the code.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">For me, the last point mentioned about black box is very interesting: a coverage criterion with a very black-box vision can actually improve upon white-box testing in terms of coverage.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Additionally, there are other kinds of test coverage that are not related to lines of code or inputting test data. One thing we must cover is mobile fragmentation: are we covering the main mobile devices, operating systems, and screen sizes? When it comes to browsers and operating systems, we must consider how will our web system behaves in any combination of operating systems and browsers and how many combinations we should try. Lastly, think about the test environment, the context, etc.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Laying_Out_the_Plan\"><\/span>Laying Out\u00a0the Plan<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">What happens when you never have enough time to reach certain criteria for your test cycles? Here I\u2019ll present a solution that works well in these instances. I have used it before and I hope it makes sense for you too.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">To illustrate the main idea, say we have different features to test on different browsers and we have organized different test cases with different test suites, each one with its own priority. We need to execute the most critical against all browsers, but the rest, we can decide to execute on a different browser. In the following test cycles, we can exchange all pairs (suite\/browser). That way, in each test cycle we do not have great coverage, but after multiple test cycles, we improve it. We cannot ever be assured that we are done with testing, but when time is scarce, we have to use it wisely and do our best to reduce risk.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Here\u2019s an example of how to plan good test coverage over many test cycles.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\"><img decoding=\"async\" class=\"aligncenter wp-image-6384\" src=\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2016\/05\/Graphs-01-1-1024x797.png\" alt=\"optimize test coverage\" width=\"500\" height=\"389\" \/><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Where it says \u201cdate 1\u201d, it could also\u00a0say \u201csprint 1\u201d, \u201citeration 1\u201d, \u201cday 1\u201d, \u201cversion 1\u201d, etc. The goal here is to distinguish which test cases you will execute in each iteration in each environment. For some of them, it is mandatory to execute test every time on all browsers (probably the most critical ones). Others can be divided into groups and executed only in one browser, but in a very clever way in order so that each is executed in each browser by the 4th execution.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Here is another example applied to mobile testing in order to reduce risk related to device fragmentation.<\/span><\/p>\n<p><span style=\"line-height: 1.71429; font-size: 1rem; color: #333333;\"><br \/>\n<img decoding=\"async\" class=\"aligncenter wp-image-6386\" src=\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2016\/05\/Graphs-02-1-1024x379.png\" alt=\"optimizing long term mobile test coverage\" width=\"800\" height=\"296\" \/><br \/>\n<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\"><span style=\"line-height: 24.0001px;\">After the third execution, you&#8217;d have this coverage:<\/span> <\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\"><strong><br \/>\n<a href=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/08\/Screen-Shot-2018-08-10-at-4.40.27-PM-min.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-10107\" src=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/08\/Screen-Shot-2018-08-10-at-4.40.27-PM-min.png\" alt=\"mobile test coverage scenario\" width=\"240\" height=\"217\" \/><\/a> <\/strong><\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Conclusion\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">Test coverage criteria are very useful, but be careful with them because they do not guarantee anything. Some criteria are linked to others, when you forget one, you forget the rest and vice versa. We need to use the ones that best suit our needs and also consider priorities for each module and define coverage to look at each one according to priority and complexity. Finally, we can apply long-term coverage criteria to optimize test coverage over time.<\/span><\/p>\n<hr \/>\n<h2><span class=\"ez-toc-section\" id=\"Recommended_for_You\"><\/span>Recommended for You<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a href=\"http:\/\/abstracta.us\/blog\/test-automation\/they-say-automation-increases-test-coverage-but-what-is-test-coverage\/\" target=\"_blank\" rel=\"noopener\">They Say Automation Increases Test Coverage, But What is Test Coverage Exactly?<\/a><br \/>\n<a href=\"http:\/\/abstracta.us\/blog\/test-automation\/the-4-most-common-test-automation-challenges-and-how-to-overcome-them\/\" target=\"_blank\" rel=\"noopener\">The 4 Most Common Challenges in Test Automation (And How to Overcome Them)<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Our method for improving test coverage over multiple test cycles We want to test as much code as humanly (or mechanically) possible right? Yes and no. For each test cycle, it\u2019s important to consider multiple strategies for measuring test coverage and to put a system&#8230;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[79,60],"tags":[65],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Optimize Test Coverage in the Long-Term | Abstracta<\/title>\n<meta name=\"description\" content=\"What happens when you never have enough time to reach certain criteria for your test cycles? Here&#039;s how to optimize test coverage in the long-term.\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Optimize Test Coverage in the Long-Term | Abstracta\" \/>\n<meta property=\"og:description\" content=\"What happens when you never have enough time to reach certain criteria for your test cycles? Here&#039;s how to optimize test coverage in the long-term.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-05-23T17:10:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:21:16+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2016\/05\/Optimize_Test_Coverage-compressor.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"420\" \/>\n\t<meta property=\"og:image:height\" content=\"236\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@fltoledo\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2016\/05\/Graphs-01-1-1024x797.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\",\"name\":\"How to Optimize Test Coverage in the Long-Term | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/#primaryimage\"},\"datePublished\":\"2016-05-23T17:10:05+00:00\",\"dateModified\":\"2025-05-05T21:21:16+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\"},\"description\":\"What happens when you never have enough time to reach certain criteria for your test cycles? Here's how to optimize test coverage in the long-term.\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/\",\"name\":\"Mobile Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/optimize-test-coverage-long-term\/\",\"name\":\"How to Optimize Test Coverage in the Long-Term\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\",\"name\":\"Federico Toledo, Chief Quality Officer at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6de7ec6536c4028b5c02ad4ec1b9af0d?s=96&d=blank&r=g\",\"caption\":\"Federico Toledo, Chief Quality Officer at Abstracta\"},\"description\":\"Co-founder and COO of Abstracta\",\"sameAs\":[\"https:\/\/twitter.com\/fltoledo\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/6343"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=6343"}],"version-history":[{"count":13,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/6343\/revisions"}],"predecessor-version":[{"id":17528,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/6343\/revisions\/17528"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=6343"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=6343"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=6343"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}