Is test automation just a huge expense or an investment?
Is there a positive ROI of test automation? Generally, it’s said that automated testing can extend the coverage and scope of testing, reduce costs, mitigate risk, put the focus on manual testing where it’s really necessary, find defects earlier, etc. etc. This seems to always be the same spiel that tool vendors pose without much proof.
While many testers themselves know that there’s a multitude of benefits associated with test automation, one of the number one challenges of getting started is obtaining management buy-in and commitment. Luckily, there is a way to convince managers that there’s something in it for them as well. Today we’ll discuss not only the ways in which test automation is valuable for software development, but also the ways in which it provides managers and financial executives real business value.
When Test Automation is Useful
The best uses for test automation involve tests that repeatable and are necessary to be run many times (either because the app will have many versions, for maintenance reasons, or because it must be tested on different platforms). One such example of this type of test is regression tests. Other tests that should be considered for automation include those that cover high-risk functionalities that, if they were to fail, would cause a great negative impact to the business.
For more on which tests to automate, read this post.
Manual vs Automated Testing
Not everything can be automated, and there’s no need to try to replace all manual testing because there are things that just can’t be automated and are not worth automating. Testing a new functionality manually allows you to quickly know more about the application at a low cost. As knowledge accumulates, the inventory of tests increases, and consequently, the cost also increases of manual testing. On the other hand, automation has a higher initial cost which decreases as it progresses. This behavior can be seen in the following graph (taken from here):
Benefits
Automated tests run on average 5 times faster (probably many more times than that, but we’ll be conservative here) than manual tests. If we automate the tests of a development cycle, we’ll be able to re-test with minimal effort during the next cycle, allowing us to increase the size of the test suite, thus expanding coverage. Manual testing lets you find the few critical defects that are easy to locate while automated testing lets you go beyond that to find subtle defects that you are unlikely to find. Lastly, If we don’t automate, we may end up with test cycles that are longer than development cycles, or decide not to test certain things at all, subjecting us to risk.
What’s the ROI of Test Automation?
A manual tester runs tests 8 hours per day and goes home. At that time, testing stops. With test automation, we can run tests 16 more hours a day (in the best case scenario… of course) for the same cost, reducing the average cost of testing hours.
Here, we will look at the case example from Paul Grossman’s white paper, “Automated Testing ROI: Fact or Fiction?”
Consider the case of practicing manual testing only. If a tester on average costs $50 an hour and if a senior tester who creates automated tests costs $75 an hour, that would cost about $400 and $600 respectively per day per tester.
Now, consider a team of 10 testers, five senior-level and five entry-level, with a monthly loaded cost of $105,000 (for 168 hours per month). We’d get a total of 1,350 hours costing $78.00/ hour (this is assuming each tester realistically works 135 hours per month due to breaks, training days, vacations, etc.). If we automate testing, the cost of labor would remain the same, but for the effort of 3 test automation engineers, we’d achieve 16 hours a day of testing and will run 5x more tests per hour. This results in the equivalent of 5,040 hours per month of manual testing created by the three test automation engineers. Then, consider the rest of the team doing manual testing (7 people x 135 hours/month). That amounts to 945 hours more, ending with a combined total of 5,985 hours of testing at $17,54/hour ($105,000 divided by 5,985 hours).
In this scenario, we’ve dramatically reduced the cost of each test hour from $78 to $17.54, which is a benefit that the CFO will clearly understand. Or you could look at it this way; we have increased testing from 1,350 hours to 5,985 equivalent hours and gained $315,000 worth of testing per month for the same cost (5,040 times the average hourly cost of a tester).
Not only do we test quicker, but the test coverage is larger, which means we can find more bugs! But, finding bugs certainly means we will have more work to do and boatloads more money needed to fix them right? Not necessarily.
It costs much less to fix bugs that are detected earlier in the development cycle. In the chart below, you can see the cost of correcting a defect detected by the stage in which it has been found (development, integration, beta testing, or production). We will assume that it costs $75/hour to fix bugs. These bug costs don’t include hidden ones as well such as loss of reputation, trust, and even equipment wear.
Data Source: (Planning Report 02-3, “The Economic Impacts of Inadequate Infrastructure for Software Testing,” Prepared by RTI for National Institute of Standards and Technology, May 2002, p 7-12.).
As you can see, the sooner we find bugs, the cheaper and easier it is to fix them. If we practice test automation, it’s more probable that we will find more bugs before the beta testing and production phases. It’s difficult to estimate how much, but in general for every bug that we find in the early stages, we will save $200 (not bad)! Coding defects found post-release cost 5x more to fix than those found during unit testing.
It’s safe to say that there is a high ROI of test automation and that it is a GOOD investment because it provides value in two ways:
Business Value
- Improve software quality
- Avoid operational problems
- Maintain a good customer image
- Avoid legal problems
- Decrease cost of fixing bugs by 5x
IT Value
- Simplifies routine tasks
- Run more tests without increasing costs in the same amount of time
- Increase the scope of coverage
- Find the hard-to-detect defects earlier, when they are easier to fix
- Improve Software Quality
Do you agree that the ROI of test automation is significant? Comment below!
Download our ebook, A Complete Introduction to Functional Test Automation:
Recommended for You
The Complete Beginner’s Guide to Functional Test Automation
The 4 Most Common Test Automation Challenges (And How to Overcome Them)
How to Avoid False Positives and False Negatives in Test Automation
Make sure you can trust the results of your test automation When dealing with automation, one of its most delicate subjects is the results that lie, otherwise known as false positives and false negatives. Those who have already automated know this to be an issue…
Automated Accessibility Testing: Comparing axe + WDIO and Pa11y-ci
Let’s dive into accessibility testing! We will explore efficient and effective tools that enhance website accessibility. Our goal is to evaluate their capabilities and coverage based on accessibility standards and learn what the best tool for which task is. Get insights and strategies for incorporating…
Abstracta
Test automation can be a truly cost-effective process that brings the best results, with the new age test automation tools like QARA Enterprise, TestComplete and Ranorex, to name a few.
Great post. Thank you so much for sharing.
A really insightful blog post on the ROI aspect of investing in an automation test tool. I’ve used quite a few of them, like QARA Test and TestComplete, and I believe that even though switching to automation can be a significant investment, it proves to be extremely worthwhile. After all, test automation management tools are all you need to reach your testing goals and save money at the same time.
If you are not looking to dedicate a substantial amount of time/resources on code-heavy automated tests, take a look at Test Studio by Telerik. https://www.telerik.com/teststudio.
Just remember that testing costs will have to come from somewhere. Even with a significant ROI that automation offers, open source still requires engineering time to create and maintain these scripts.
Hi, Is anyone usign Katalon studio for automation? We have doen POC and seems to be workign great, but I would like to get feedback before we fully implement.
HI Suji,
I have tried Katalon and have been using it for multiple PoC’s in my current gig. Works great with browser automation on desktops but have a lot of dependencies to run the automation tests on mobile devices.
Hey Sophia, Good write up, Would we be able to calculate Automation Leverage?
Who is testing the Test Automation. Too many times I am finding people who are automating that don’t either understand the application, or don’t take the time to test their scripts. If you do automation, please take the time to do code walkthroughs and review your test coverage.
Good article but truly not that black and white with Test Automation. For larger organizations where there 100s of Testing resources, the ROI calculation cannot be focused only on the Tests cases automated. Two elements to be considered would the effectiveness and efficiency of the tests. We could be doing 50 builds and in multiple environments, by pure execution calculations a simple $100 K investment could return you millions. The Test coverage plays into the effectiveness of the tests. A test needs to uncover something if it doesn’t then either the code is too darn good or test is irrelevant. I like the way you describe the Business Value and IT Value. Again, thank you for sharing your thoughts.
Great Article on ROI…earlier days it used to be difficult to justify automation and had to really fight for setting up automation by showing ROI. These days with Agile Automation is part of overall engineering process …I see automation has come a long way and now people are thinking automation beyond ROI which is good.
What about the cost of maintenance of automated tests? Let’s say there was a minor update to the core module, after that change it is quite possible that you will need to update/rewrite large part of automated tests.
Hi K, that’s one of the reasons why the approach to automation is critical to get the higher ROI. Check out this other post: https://abstracta.us/blog/test-automation/best-testing-practices-for-agile-teams-the-automation-pyramid/
Unfortunately that post isn’t available anymore. Any chance to get it republished?
Hi! To which post are you referring? Thanks!
This one: https://abstracta.us/blog/test-automation/best-testing-practices-for-agile-teams-the-automation-pyramid/
Hi Sofia,
Your article is great!! I am currently doing a cost-benefit analysis of test automation for ROI. Thank you for sharing this. I hope that you will grant me permission to cite you and some parts of this article.
Yes, feel free! Thanks Arvie, I am glad you found it useful.
Great article!
What tool do you recommend for API testing?
Hi Alex! Have you tried SoapUI or Rest Assured?
Great Article. But I Wondering for Which Tool i use for Testing and its should be a open source, because company’s not spend money on that…
Hi Somnath!
I recommend Selenium for functional test automation and
Jmeter for automation of performance tests – both are open source. We use them ourselves at Abstracta.
🙂
Great article! One aspect that I miss is the fact that testers spend less time executing tests and thus have more time to focus on quality of tests. I know this is hard to put real numbers on and will vary for each company, but it’s worth mentioning.
Ruben, totally agree !
You are talking only about quantitative. What about qualitative? There is a huge different between the automated and manual testing results.
Ref.
https://en.wikipedia.org/wiki/Quantitative_research
https://en.wikipedia.org/wiki/Qualitative_research
Hi Ricardo,
Yes, there is indeed. And also the quality of the automated tests is an important part.
Please, feel free to develop more this idea. Would like to see your thoughts.
Regards,
Sofia