The 4 Most Common Test Automation Challenges (and How to Overcome Them)
Automation testing is no walk in the park, but here are some tips to pull it off effectively
“Less than half of the IT projects worldwide are successful partly because of inefficient testing practices.” -World Quality Report 2014-2015
They say “automating chaos just gives you faster chaos,” not only faster, but (to paraphrase a song by Daft Punk) harder, faster, stronger… chaos. Automation can be an awesome productivity booster within testing teams and a quality enhancer for your systems when used correctly. The key is to make sure that it is used the right way, which is the hardest part when starting out. Here we will look at the most common test automation challenges and how to overcome them.
Note: Much of this information is covered in great detail in Randall Rice’s Surviving the Top Ten Challenges of Test Automation and from Federico Toledo’s book, Introduction to Information Systems Testing.
Challenge 1: Receiving the Green Light from Management
As with any company department, associates are always asking for things that may or may not be allowed for in the budget. Testers may already know that automation offers both business and IT benefits (decrease time to market, increase test coverage and accuracy, decrease cost per hour, find bugs sooner, etc), but how can testers convince the finance department and the QA director to allocate the necessary time and funds for implementing test automation?
“Test automation promises increased productivity and accuracy, which is where the business case must be made. The cost of a single defect … can offset the price of one or more tool licenses.” — Randall Rice
We agree with Randall Rice here that automation can pay for itself. To prove to management that the financial benefits are substantial, you can show them this simple breakdown we did of the ROI of test automation. In it we demonstrated how a team of 5 senior and 5 entry level testers could hypothetically reduce the cost of testing from $78/hour to just $17.58/hour and increase testing from 1,350 hours per month to 5,985 equivalent hours, gaining $315,000 worth of testing via automation. Not to mention all of the qualitative benefits of automation.
It is important to also be very transparent with any and all stakeholders. Don’t lie to them and say that automation doesn’t require much effort up front, because it surely does, but in the end, it’s worth it!
Challenge 2: Selecting and Using the Appropriate Tools
Many teams do not get past this phase due to several reasons. They may lack the expertise to use a certain tool, the tool they want doesn’t exist, the tool or set of tools do not offer 100% test case coverage, the cost of a tool exceeds the test budget, etc.
“54% of IT leaders indicate that their organizations lack suitable tools for automation while provisioning test environments to their teams.” -World Quality Report 2014-2015
If you don’t have a sufficient base knowledge for how to use a tool, you have a few options:
- Take an online course
- Have someone that helped create the tool come and give training sessions
- Hire a consultant to help you master it
- When all else fails, outsource (Read our white paper: 10 Mistakes Companies Make When Outsourcing Software Testing)! It may be quicker to simply hire someone who already has the expertise to use it and employ him or her for your automation project.
If you think a tool doesn’t exist, it might be good to confirm it with the testing community. Go on to forums like Stack Exchange where fellow testers love to discuss developments in testing. If you can’t find it, you might want to see if it’s feasible or worthwhile to create it yourself. In our case, we have created several tools that we have made available to the open source community on our Abstracta Github account.
If the tool you have doesn’t do everything you need, consider finding a multi-tool solution. Remember, it’s impossible to test absolutely everything, but you can use the tools that test the most important things.
Lastly, If a tool is out of the budget, do a quick cost vs. benefit analysis and present your case. You can measure the damage done by a previous bug you have encountered and show how much time and money you could have saved if you had the tool in place. Again, you can also present them the information in our post, the ROI of Test Automation.
Challenge 3: Identifying a Starting Strategy
Ok, so you might have all the tools and the support to begin automating, but what do you actually automate and how? Unfortunately, the tools themselves do not tell you what to automate, just as new parents find to their dismay that children do not come with a handbook. Will you raise a generation of outstanding automated tests or will they turn out to be spoiled wrecks? Of course you’d hope for the former! In reality, you can’t automate everything so you have to be strategic. You can use two approaches to help with this: risk-based testing and the automation pyramid.
Risk-based testing gives higher priority to testing the elements that are most at risk of failing which also carry the greatest negative consequences if said failure occurs. Here you should take into consideration:
- The financial impact of potential errors
- Probability of failure (here it is a good idea to ask the developers what they think)
- Service level agreements (SLA)
- Is there money or are there lives at stake? (Yes, this may seem dramatic, but there are several systems that deal with such important matters.)
This should give you a good method for prioritizing which test cases to automate. Another approach that we highly recommend is to follow the automation pyramid. We wrote much more extensively about this topic in a recent post, but here is a quick overview.
The ice cream cone is sweet and tempting, but it could spoil your appetite for automation! Following the ice cream cone approach will result in high levels of frustration because it emphasizes automation on the UI level, which employs more brittle tests that break easily. Whereas, if you focus on automated unit tests, you are helping to prevent bugs or eliminate them almost immediately as you go through the software development life cycle.
Challenge 4: Setting Realistic Expectations of Automation
No matter how great your tools and processes are, it’s important to remember that testing is never complete. Test automation is not a panacea for bug laden systems and shouldn’t be used in place of, but in conjunction with non-automated tests. There are some tests that simply can’t be automated, but there are some automated tests that uncover bugs which couldn’t be found otherwise.
We agree with the test automation philosophy of James Bach and Michael Bolton. Test automation is really just automatically checking the system, while humans are still needed to do the non-automated testing. Also, remember that the value of a test comes from the information that it provides, not the quantity of tests executed, nor the frequency. What we care most about is if we are getting the right information so that we can make the best possible decisions when improving the quality of our systems.
Make sure your team and management agree on and understand the desired outcome(s) from your automation plan so that everyone is on the same page!
Thanks for reading! I hope this helps you automate tests more effectively. Have you faced any other test automation challenges and what you did to overcome them?