The True ROI of Test Automation
Is test automation just a huge expense or an investment?
Is there a positive ROI of test automation? Generally, it’s said that automated testing can extend the coverage and scope of testing, reduce costs, mitigate risk, put the focus on manual testing where it’s really necessary, find defects earlier, etc. etc. This seems to always be the same spiel that tool vendors pose without much proof.
While many testers themselves know that there’s a multitude of benefits associated with test automation, one of the number one challenges of getting started is obtaining management buy-in and commitment. Luckily, there is a way to convince managers that there’s something in it for them as well. Today we’ll discuss not only the ways in which test automation is valuable for IT, but also the ways in which it provides managers and financial executives real business value.
When Test Automation is Useful
The best applications for implementing automated testing are when the tests are repeatable and it’s necessary to run them many times (either because the app will have many versions, maintenance reasons, or because it must be tested on different platforms). This is also known as regression testing.
Manual vs Automated Testing
Not everything can be automated, and there’s no need to try to replace all manual testing because there are things that just can’t be automated. For example, if visual verification is needed, manual testing is in order. Sometimes it is simply easier to run something manually than automatically. So how to balance between the two? Each case will be different, but typically, it’s good to try to start with a target of 10 to 15% regression testing, and as you go along, you may go as high as about 60%.
Automated tests run on average 5 times faster (probably many more times than that, but we’ll be conservative here) than manual tests. If we automate the tests of a development cycle, we’ll be able to re-test with minimal effort during the next cycle, allowing us to increase the size of the test suite, thus expanding coverage. Manual testing lets you find the few critical defects that are easy to locate while automated testing lets you go beyond that to find subtle defects that you are unlikely to find. Lastly, If we don’t automate, we may end up with test cycles that are longer than development cycles, or decide not to test certain things at all, subjecting us to risk.
What’s the ROI of Test Automation?
(This is the part that you copy, paste, and hit “send” to the finance dept!)
A manual tester runs tests 8 hours per day and goes home. At that time, testing stops. With test automation, we can run tests 16 more hours a day (in the best case scenario… of course) for the same cost, reducing the average cost of testing hours.
Here, we will look at the case example from Paul Grossman’s white paper, “Automated Testing ROI: Fact or Fiction?”
Consider the case of practicing manual testing only. If a tester on average costs $50 an hour and if a senior tester who creates automated tests costs $75 an hour, that would cost about $400 and $600 respectively per day per tester.
Now, consider a team of 10 testers, five senior-level and five entry-level, with a monthly loaded cost of $105,000 (for 168 hours per month). We’d get a total of 1,350 hours costing $78.00/ hour (this is assuming each tester realistically works 135 hours per month due to breaks, training days, vacations, etc.). If we automate testing, the cost of labor would remain the same, but for the effort of 3 test automation engineers, we’d achieve 16 hours a day of testing and will run 5x more tests per hour. This results in the equivalent of 5,040 hours per month of manual testing created by the three test automation engineers. Then, consider the rest of the team doing manual testing (7 people x 135 hours/month). That amounts to 945 hours more, ending with a combined total of 5,985 hours of testing at $17,54/hour ($105,000 divided by 5,985 hours).
In this scenario, we’ve dramatically reduced the cost of each test hour from $78 to $17.54, which is a benefit that the CFO will clearly understand. Or you could look at it this way; we have increased testing from 1,350 hours to 5,985 equivalent hours and gained $315,000 worth of testing per month for the same cost (5,040 times the average hourly cost of a tester).
Not only do we test quicker, but the test coverage is larger, which means we can find more bugs! But, finding bugs certainly means we will have more work to do and boatloads more money needed to fix them right? Not necessarily.
It costs much less to fix bugs that are detected earlier in the development cycle. In the chart below, you can see the cost of correcting a defect detected by the stage in which it has been found (development, integration, beta testing, or production). We will assume that it costs $75/hour to fix bugs. These bug costs don’t include hidden ones as well such as loss of reputation, trust, and even equipment wear.
Data Source: (Planning Report 02-3, “The Economic Impacts of Inadequate Infrastructure for Software Testing,” Prepared by RTI for National Institute of Standards and Technology, May 2002, p 7-12.).
As you can see, the sooner we find bugs, the cheaper and easier it is to fix them. If we practice test automation, it’s more probable that we will find more bugs before the beta testing and production phases. It’s difficult to estimate how much, but in general for every bug that we find in the early stages, we will save $200 (not bad)! Coding defects found post-release cost 5x more to fix than those found during unit testing.
It’s safe to say that there is a high ROI of test automation and that it is a GOOD investment because it provides value in two ways:
- Improve software quality
- Avoid operational problems
- Maintain a good customer image
- Avoid legal problems
- Decrease cost of fixing bugs by 5x
- Simplifies routine tasks
- Run more tests without increasing costs in the same amount of time
- Increase scope of coverage
- Find the hard-to-detect defects earlier, when they are easier to fix
- Improve Software Quality
Do you agree that the ROI of test automation is significant? Comment below!