{"id":5274,"date":"2016-04-04T17:53:29","date_gmt":"2016-04-04T17:53:29","guid":{"rendered":"http:\/\/www.abstracta.us\/?p=5274"},"modified":"2025-05-05T21:21:20","modified_gmt":"2025-05-05T21:21:20","slug":"avoid-false-positives-false-negatives-test-automation","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/","title":{"rendered":"How to Avoid False Positives and False Negatives in Test Automation"},"content":{"rendered":"<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n<h1><span style=\"font-weight: 400; color: #333333;\">Make sure you can trust the results of your test automation<\/span><\/h1>\n<p><span style=\"font-weight: 400; color: #333333;\">When dealing with automation, one of its most delicate subjects is the results that lie, otherwise known as <i>false positives <\/i>and <i>false negatives<\/i>. Those who have already automated know this to be an issue and those who are about to begin, let us give you fair warning that you will encounter this problem. What can we do to <strong>avoid false positives and negatives in test automation<\/strong>? What can we do so that the test case does what it is supposed to do? Doesn\u2019t that sound like testing?<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">These definitions come from the medical field:<\/span><\/p>\n<ul>\n<li><span style=\"color: #333333;\"><b>False Positive:<\/b><span style=\"font-weight: 400;\"> an examination indicates a disease when there is none.<\/span><\/span><\/li>\n<li><span style=\"color: #333333;\"><b>False Negative:<\/b><span style=\"font-weight: 400;\"> an examination indicates everything is normal when in fact the patient is sick.\u00a0<\/span><\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400; color: #333333;\">If one were to translate this to our field, we could say the following:<\/span><\/p>\n<ul>\n<li><span style=\"color: #333333;\"><span style=\"font-weight: 400;\"><strong>False Positive:<\/strong> when a test is executed and despite it running correctly, the test tells us there is an error (that there is a disease). This adds a lot of cost, as the tester will search for the non-existent bug.<\/span><\/span><\/li>\n<li><span style=\"color: #333333;\"><span style=\"font-weight: 400;\"><strong>False Negative:<\/strong> when the execution of a test shows no faults even though there is a bug in the application. This, as much as the false positive, can be due to an incorrect initial state of the database or problems dealing with the test environment setting.<\/span><\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400; color: #333333;\">If we believe that the false positive is a problem due to the extra costs, with a false negative, errors are there but we are not aware of them and we feel at ease! We trust all functionalities are covered and that they are being tested, therefore they must not have any mistakes.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">We obviously want to avoid results lying to us! No one likes lies. Automated test case results are expected to be\u00a0reliable so that we can be assured that we aren&#8217;t wasting time checking whether the results are correct or not.<\/span><\/p>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">The only choice is to carry out a proactive analysis, checking the quality of our tests and anticipating possible errors. We must be actually thinking about the test and not simply doing a <\/span><i><span style=\"font-weight: 400;\">record and playback<\/span><\/i><span style=\"font-weight: 400;\">.<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">To lower the risk of environment or data problems, we should have a controlled environment only accessible through automated tests. With this, we are already avoiding some major headaches because if the data is constantly changing, we will not be able to reproduce problems detected by the tests and we won&#8217;t be able to find out what&#8217;s the matter with them.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Moreover, we should check the actual test cases! Because who can assure us they are programmed correctly? <\/span><\/p>\n<p><strong><span style=\"color: #333333;\">[tweet_box design=&#8221;default&#8221; float=&#8221;none&#8221;]At the end of the day, the test code is code after all, and as such, can exhibit flaws or be improved.[\/tweet_box]<\/span><\/strong><\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">And who better than us testers to test them?<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"In_Search_of_False_Positives\"><\/span>In Search of False Positives<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">If the software is &#8220;healthy,&#8221; and we don&#8217;t want it to display any errors, we must make sure the test is testing what it wants to test. So, we must verify the starting conditions just as much as the final ones. Meaning, a test case tries to execute a determined set of actions with certain input data to verify the outgoing data and the final state, but it is highly important (especially when the system we are testing uses a database) to make sure the initial state is what we expected it to be.<\/span><\/p>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">Therefore, if for example, we are creating an instance of a particular entity in the system, the test should verify if that information already exists before beginning the execution of the actions to be tested, because if so, the test will fail (due to duplicate key or similar) but in reality the problem is not with the system but with the data on the test. We have two options: checking if it exists, and if so, we&#8217;ve already used that information, or we finish off the test by saying the result is \u201cinconclusive\u201d (or are <\/span><i><span style=\"font-weight: 400;\">pass<\/span><\/i><span style=\"font-weight: 400;\"> and <\/span><i><span style=\"font-weight: 400;\">fail<\/span><\/i><span style=\"font-weight: 400;\"> the only possible results for a test?).<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">If we make sure all the things that could affect our result are in place, just as expected, then we will reduce the percentage of errors that aren&#8217;t errors.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"In_Search_of_False_Negatives\"><\/span>In Search of False Negatives<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">If the software is &#8220;sick,&#8221; the test must fail! One way of detecting false negatives is to insert errors into the software and verify that the test case finds the mistake. This goes in line with mutation testing. It is very difficult when not working directly with the developer to input the mistakes into the system. It\u2019s also quite expensive to prepare every error, compile it and deploy it, and so on, and to verify that the test finds that fault. In many cases, it can be done by varying the data of the test or playing around with different things. For example, if I have a plain text file as input, I can change something in the content of the file in order to force the test to fail and verify that the automated test case finds that error. In a parameterizable application, it could also be achieved by modifying some parameter.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #000000;\"><span style=\"color: #333333;\">The idea is to verify that the test case realizes the mistake and that&#8217;s why we try to make it fail with these alterations. Anyway, what we could at least do is think about what happens if the software fails at this point, will this test case notice it, or should we add some other<\/span> <span style=\"text-decoration: underline; color: #00b674;\"><span style=\"text-decoration: underline;\"><a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/jmeter-response-assertions-how\/\">validation<\/a><\/span><\/span><span style=\"color: #333333;\">?<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Both strategies will allow us to have more robust test cases, but keep in mind: would they be more difficult to keep up later? Of course, this will not be done to every test case we automate, only to the most critical ones, or the ones really worthwhile, or perhaps the ones we know will stir up trouble for us every now and again.<\/span><\/p>\n<h4>What do you do to prevent false positives and false negatives in test automation?<\/h4>\n<hr \/>\n<h2><span class=\"ez-toc-section\" id=\"Recommended_for_You\"><\/span><strong>Recommended for You<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/jmeter-response-assertions-how\/\">JMeter Response Assertions: How to know what to validate in an HTTP Response Request<\/a><br \/>\n<a href=\"http:\/\/abstracta.us\/blog\/test-automation\/the-4-most-common-test-automation-challenges-and-how-to-overcome-them\/\">The 4 Most Common Test Automation Challenges (and How to Overcome Them)<\/a><\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Make sure you can trust the results of your test automation When dealing with automation, one of its most delicate subjects is the results that lie, otherwise known as false positives and false negatives. Those who have already automated know this to be an issue&#8230;<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60],"tags":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to Avoid False Positives and False Negatives in Test Automation | Abstracta<\/title>\n<meta name=\"description\" content=\"What can we do to\u00a0avoid false positives and negatives in test automation? What can we do so that the test case does what it is supposed to do?\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to Avoid False Positives and False Negatives in Test Automation | Abstracta\" \/>\n<meta property=\"og:description\" content=\"What can we do to\u00a0avoid false positives and negatives in test automation? What can we do so that the test case does what it is supposed to do?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2016-04-04T17:53:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:21:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2016\/04\/Avoid_false-min-compressor.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"420\" \/>\n\t<meta property=\"og:image:height\" content=\"236\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AbstractaUS\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\",\"name\":\"How to Avoid False Positives and False Negatives in Test Automation | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"datePublished\":\"2016-04-04T17:53:29+00:00\",\"dateModified\":\"2025-05-05T21:21:20+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/78cd0dcae50ce820b25e86d3330e9762\"},\"description\":\"What can we do to\\u00a0avoid false positives and negatives in test automation? What can we do so that the test case does what it is supposed to do?\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/\",\"name\":\"Test Automation\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/avoid-false-positives-false-negatives-test-automation\/\",\"name\":\"How to Avoid False Positives and False Negatives in Test Automation\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/78cd0dcae50ce820b25e86d3330e9762\",\"name\":\"Sof\\u00eda Palamarchuk, Co-CEO at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/222e8b1136482564fe25acc4de2b9b7a?s=96&d=blank&r=g\",\"caption\":\"Sof\\u00eda Palamarchuk, Co-CEO at Abstracta\"},\"description\":\"Co-Chief Executive Officer at Abstracta\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/5274"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=5274"}],"version-history":[{"count":13,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/5274\/revisions"}],"predecessor-version":[{"id":10838,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/5274\/revisions\/10838"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=5274"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=5274"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=5274"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}