{"id":9606,"date":"2017-12-10T00:33:18","date_gmt":"2017-12-10T00:33:18","guid":{"rendered":"http:\/\/abstracta.us\/blog\/?p=9606"},"modified":"2025-05-05T21:20:40","modified_gmt":"2025-05-05T21:20:40","slug":"losing-regaining-confidence-automated-tests","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/","title":{"rendered":"Losing and Regaining Confidence in Automated Tests"},"content":{"rendered":"<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n<h1><span style=\"font-weight: 400;\">Why we lose faith in automated tests and\u00a0what\u00a0to do about it<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">Every now and then we find ourselves meeting clients who have lost faith in the usefulness of automated tests. Not necessarily because of the quality of the tests themselves, but because many times they\u2019re unable to clearly and continuously convey the value of the tests and what they contribute to the project. In this post, we\u2019ll share some strategies that we have used for regaining confidence in automated tests on our part and the client\u2019s.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"When_is_the_Confidence_Lost\"><\/span><strong><span style=\"color: #00b674;\">When is the Confidence Lost?<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400;\">In some situations, we have seen a team come to question whether or not to continue with their automated tests. <\/span><b>Having already invested the time and effort to prepare the tests, they must decide if they want to continue investing in maintaining them or toss them out.<\/b><\/p>\n<p><span style=\"font-weight: 400;\">But how does a team get to that point? <\/span><\/p>\n<p><span style=\"font-weight: 400;\">A common factor in all these projects is that the automation work focuses more on the <\/span><i><span style=\"font-weight: 400;\">creation<\/span><\/i><span style=\"font-weight: 400;\"> of test cases than on frequently analyzing the value of this work and aligning it with the expectations of the team and what the business needs each moment. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Many times, the goal of automated testing is something like, &#8220;We want to automate the entire user interface.&#8221; Going about it that way, as the weeks pass, the set of test cases will increase, thus increasing the time required to maintain the automation, analyze the results, etc. This is even worse if all sorts of test cases are implemented, from simple to complex cases, difficult ones to maintain, etc.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For example, we participated in a project in which the team ended up with a set of more than 600 automated test cases. They weren\u2019t clear about the utility of each one, and with such a high cost of maintaining them, the tests often produced false positives because they weren\u2019t adjusted often enough. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Another complexity that the \u201clet\u2019s automate everything\u201d approach presents is the dynamics it generates. You eventually see automation as a project apart from development and not as an integrated activity aligned with the same objectives. It can happen so that without maintenance, the first automated test cases begin to become obsolete. This is often difficult because of the desire to automate everything, without giving rise to the work that is necessary to maintain a healthy and stable test architecture.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">What happens in the face of all this? Well,<\/span><b> confidence in automation is lost.<\/b><span style=\"font-weight: 400;\"> In this situation, it\u2019s best to pause the automation to define a strategy to reverse the situation.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Lessons_Learned\"><\/span><strong><span style=\"color: #00b674;\">Lessons Learned<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400;\">From experiences in which we had to &#8220;fight&#8221; for regaining confidence in automatic tests, we\u2019ll share some of the interesting lessons we learned and strategies that proved useful.<\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Know_the_Objectives_and_Prioritize_Accordingly\"><\/span><strong><span style=\"color: #3056a2;\">Know the Objectives and Prioritize Accordingly<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">Perhaps one of the most important lessons: always aim to generate value. We must always keep in mind the objectives of the project and the business itself in order to adjust the objectives of the automation project. One technique that has been of great help to align us in the objectives is the use of MindMaps. They help us to outline the modules and functionalities of the system to be tested and thus, in a graphic way, order priorities and define a plan.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In order to generate value, it is extremely necessary to be one more member of the team, and not to make the automation project something that runs parallel and independent of development. Something that has been positive for us is <\/span><a href=\"https:\/\/abstracta.us\/blog\/software-testing\/can-testers-scrum\/\"><span style=\"font-weight: 400;\">getting involved in the framework of the client\u2019s &#8220;Scrum&#8221; team<\/span><\/a><span style=\"font-weight: 400;\">, participating in daily meetings and other events, aligning day by day. By participating in meetings like these, where the whole team is involved, it\u2019s possible to understand the priorities of the project as well as the functionalities planned for each Sprint. As expected, this information is valuable input to prioritize automation work.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">One crucial thing to remember is to distinguish the ROI of the automation of the different modules, according to their particularities. There will be cases where due to the technology used, environmental difficulties, data, etc., it will be so difficult to achieve reliable automation, that it\u2019s not preferable to automate. With that said, we don\u2019t recommend starting out with the objective: &#8220;automate everything.&#8221;<\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Make_Visible_the_Results_and_Value_of_the_Tests\"><\/span><strong><span style=\"color: #3056a2;\">Make Visible the Results and Value of the Tests<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">It\u2019s key to give visibility to the benefits that the tests bring to the team. This can be achieved by, as testers, giving our status in dailys so that everyone knows what we are working on, and how it affects others and the product. That way we also build up the idea that the test cycle brings something real to the product, and<\/span><i><span style=\"font-weight: 400;\"> it is a part<\/span><\/i><span style=\"font-weight: 400;\"> of the activity of development. Ideally, even, it would be great to reach the point where automation is not seen as a separate task, but is part of the \u201cDefinition of Done\u201d for the features that we want to test with automation. In this way, the team will focus on everything that is necessary for automation to be available on time, in order to complete the tasks that are committed to the Sprint.<\/span><\/p>\n<p><b>While the use of tools such as <a href=\"https:\/\/jenkins-ci.org\/\" target=\"_blank\" rel=\"noopener\">Jenkins<\/a> facilitate the visibility of the tests, giving constant results about their status, if false positives are not controlled, confidence is immediately lost, and the information reported in the tools is dismissed. <\/b><\/p>\n<p><span style=\"font-weight: 400;\">Faced with this type of situation, a possible strategy is to dedicate effort to let Jenkins run only those cases that are high priority and <\/span><b>run consistently<\/b><span style=\"font-weight: 400;\">, that only fail when errors <\/span><i><span style=\"font-weight: 400;\">actually<\/span><\/i><span style=\"font-weight: 400;\"> occur in the system. In this sense I liked a tweet from <a href=\"https:\/\/twitter.com\/katrina_tester\" target=\"_blank\" rel=\"noopener\">Katrina Clokie<\/a> regarding an Agile Testing Days 2017 session:<\/span><\/p>\n<p>&nbsp;<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\"><p>\nIf there&#8217;s a failing automated test and no one is fixing it, delete it. If people don&#8217;t care that it&#8217;s failing, they don&#8217;t care about what it&#8217;s testing. <a href=\"https:\/\/twitter.com\/PaulHolland_TWN?ref_src=twsrc%5Etfw\">@PaulHolland_TWN<\/a><a href=\"https:\/\/twitter.com\/hashtag\/AgileTD?src=hash&amp;ref_src=twsrc%5Etfw\">#AgileTD<\/a><\/p>\n<p>\u2014 Katrina Clokie (@katrina_tester) <a href=\"https:\/\/twitter.com\/katrina_tester\/status\/931078980857458688?ref_src=twsrc%5Etfw\">November 16, 2017<\/a><\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<p><span style=\"font-weight: 400;\">In a specific project where we applied this strategy, at the beginning there were approximately 80 test cases running in Jenkins, of which more than 50 failed. After making the change, we managed to have more than 50 cases running correctly in Jenkins, thus recovering confidence in the test report.<\/span><\/p>\n<p><b>The important thing here is that automation contributes to the objective of testing: to provide relevant information to all interested parties about the possible problems that may arise, the risks, and ultimately, about the quality of the system.<\/b><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Generate_Opportunities_for_Knowledge_Exchange\"><\/span><strong><span style=\"color: #3056a2;\">Generate Opportunities for Knowledge Exchange<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p><span style=\"font-weight: 400;\">It\u2019s immensely helpful to ask the opinion of other people working on automation projects and generate a space for exchange that broadens the vision with which we are working and focusing on the project. As a result of this exercise, we have always benefitted, gathering amongst everyone, including experts, a set of good practices and a combination of tools that improve the stability of the test cases.<\/span><\/p>\n<p><script src=\"https:\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Takeaways\"><\/span><strong><span style=\"color: #00b674;\">Takeaways<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400;\">In summary, what we hoped to keep from these experiences that we wanted to share with you are:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Know for what\/whom the tests are directed.<\/b><span style=\"font-weight: 400;\"> Without this, the project becomes the development of many test cases that may not actually have any business value.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Prioritize modules and functionalities<\/b><span style=\"font-weight: 400;\">. Know where to start, plan the work according to the needs of the business, and be agile in that if something new comes out with higher priority, for example, a new functionality, adapt to that.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Generate a real interest in the developers<\/b><span style=\"font-weight: 400;\"> regarding the results and the value that the tests provide. This is linked to the above, always aim at generating value, and for this it is extremely necessary to be one more member of the team, and not to make the automation project parallel and independent to development. This could be achieved by including the tasks of automation in the planning of the functionalities, that is, planning the task of automating a test as part of the \u201cDefinition of Done\u201d of the feature.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Consult with other teams<\/b><span style=\"font-weight: 400;\"> about how their automation projects are being managed to apply a proven set of good practices.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">And you, have you faced a similar situation? Have you ever had to help others (or yourself) in regaining confidence in automated tests? What strategies did you use? <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Leave us a comment!<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Thank you <a href=\"https:\/\/www.linkedin.com\/in\/matias-fornara-ernst-396075a8\/\" target=\"_blank\" rel=\"noopener\">Matias Fornara<\/a> for helping me with this post!<\/span><\/p>\n<hr \/>\n<h2><span class=\"ez-toc-section\" id=\"Recommended_for_You\"><\/span><strong>Recommended for You<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><a href=\"http:\/\/abstracta.us\/blog\/software-testing\/can-testers-scrum\/\"><span style=\"font-weight: 400;\">Can There be Testers in Scrum?<\/span><\/a><br \/>\n<a href=\"http:\/\/abstracta.us\/blog\/test-automation\/the-true-roi-of-test-automation\/\"><span style=\"font-weight: 400;\">The True ROI of Test Automation<\/span><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Why we lose faith in automated tests and\u00a0what\u00a0to do about it Every now and then we find ourselves meeting clients who have lost faith in the usefulness of automated tests. Not necessarily because of the quality of the tests themselves, but because many times they\u2019re&#8230;<\/p>\n","protected":false},"author":55,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[60],"tags":[68,641,223,107],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Losing and Regaining Confidence in Automated Tests | Abstracta<\/title>\n<meta name=\"description\" content=\"In this post we\u2019ll share some strategies that we have used for regaining confidence in automated tests on our part and the client\u2019s.\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Losing and Regaining Confidence in Automated Tests | Abstracta\" \/>\n<meta property=\"og:description\" content=\"In this post we\u2019ll share some strategies that we have used for regaining confidence in automated tests on our part and the client\u2019s.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-12-10T00:33:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:20:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2017\/12\/RegainingConfidenceinAutomatedTests-min-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"560\" \/>\n\t<meta property=\"og:image:height\" content=\"315\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AbstractaUS\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\",\"name\":\"Losing and Regaining Confidence in Automated Tests | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"datePublished\":\"2017-12-10T00:33:18+00:00\",\"dateModified\":\"2025-05-05T21:20:40+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/3cc530c545cab16fae6829f65fe4419e\"},\"description\":\"In this post we\\u2019ll share some strategies that we have used for regaining confidence in automated tests on our part and the client\\u2019s.\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/\",\"name\":\"Test Automation\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\",\"url\":\"https:\/\/abstracta.us\/blog\/test-automation\/losing-regaining-confidence-automated-tests\/\",\"name\":\"Losing and Regaining Confidence in Automated Tests\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/3cc530c545cab16fae6829f65fe4419e\",\"name\":\"Abstracta Team\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6cab9c9f3dec946bd8867fdb2abbd10a?s=96&d=blank&r=g\",\"caption\":\"Abstracta Team\"},\"description\":\"We are a technology solutions company specializing in software testing, custom software development, and AI-driven software solutions. We provide top-notch, holistic solutions to enable continuous delivery of high-quality software. Our purpose is to co-create first class software, generating opportunities for development in our communities to improve people's quality of life. Organizations such as BBVA Financial Group, CA Technologies and Shutterfly turn to us for comprehensive quality solutions, from rigorous testing to innovative AI copilots and bespoke software development. Sharing our learnings with the community is rooted in our values. That is why we believe in collaborating with the IT community by sharing quality content, courses, and promoting thought leadership events. Recognized with several awards, we are committed to quality, innovation, and customer satisfaction. Our experienced team, dedicated to continuous learning and improvement, has earned the trust of numerous clients worldwide, from startups to Fortune 500 companies. We are a fast-growing company, and we are looking for proactive and talented people, who can assume responsibilities, bring new ideas, and who are as excited as we are about our mission of building high-quality software. If you are interested in joining the team, apply here https:\/\/abstracta.us\/why-us\/careers.\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/9606"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/55"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=9606"}],"version-history":[{"count":15,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/9606\/revisions"}],"predecessor-version":[{"id":16028,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/9606\/revisions\/16028"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=9606"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=9606"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=9606"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}