COVID-19’s impact on test automation, today’s biggest hurdles, and looking ahead
In June, our friends at Testim conducted a survey on end-to-end test automation. So, what were the results?
Watch this webinar recording where our COO, Federico Toledo, joined Testim CEO Oren Rubin to uncover what developers, QA, and managers are saying about test automation.
They discussed the results of the survey and compared it to other data points gathered from related industry sources and their own experiences.
- How has COVID-19 affected test automation?
- What are the top challenges and priorities for automation
- What do we expect in the future?
Watch the Webinar
TL;DW (Too Long, Didn’t Watch)?
Who Took the Survey?
To start off, here’s a breakdown of the survey respondents by role:
- 46% – Automation Engineers
- 18% – Developers / Dev managers
- 16% – QA Testers
- 13% – QA Managers
- 7% – Other
And by team size:
- 29% – 100 or more
- 78.1% – Less than 100
What’s maybe even more interesting, is how the respondents answered they organized their teams:
From our experience, we were not too surprised to see these results, since in most of our own projects, we integrate our testers with our client’s development team, whether they’re using Scrum, or kanban, etc. We see it’s highly valuable to have the testers participate in the team’s daily calls, retrospectives, plannings, and even in product demos for the customer. This helps everyone in development to remain aligned and work towards a common goal.
What Skills Are Most Critical for Testers?
On critical skills needed for testing at @testim_io webinar today: “People want to write tests fast, but we must also think about the day after and what we’re doing to maintain our tests…reuse of code or functionality are critical.”- Oren Rubin at @testim_io pic.twitter.com/9AppBqYSdP— Tristan Lombard (@TristanLombard2) October 27, 2020
Coming in at number one for most important skills was test design and architecture, which makes it seem that the respondents emphasized the need for coding skills. Oren Rubin agreed that this was very important: being able to design and implement tests is critical.
Federico also wanted to place emphasis on the need for the human side of testing, looking beyond the technical answer choices that were provided:
What Are Your Biggest Challenges in Test Automation?
“Test automation is about stability and maintenance. This is critical for your automation.” @shexman, CEO/Founder @testim_io discussing survey results for 2020. pic.twitter.com/eCTmayQglI— Tristan Lombard (@TristanLombard2) October 27, 2020
The fact that most people complained that test maintenance is one of the greatest challenges they face when it comes to test automation didn’t surprise the Testim team (given that they built a tool to help solve for that!). The next biggest pain that the survey indicated was the long time it takes to write UI functional tests.
Trends Impacting Test Automation in 2020
The top trends that respsondents indicated around test automation today were:
- CI/CD – 55.6%
- AI-assisted Testing – 46%
- Continuous Testing – 38.1%
- Shifting Left – 33.3%
- Agile Testing – 55.4%
- Testing in production – 25.4%
As Federico commented, these answers are consistent with our own experiences, as the shift-left testing approach becomes more popular, we see more engagement from different team members, taking part in automation, whether its unit testing or API testing. So these methodologies that are gaining more traction impact the way we automate and of course, this relates to CI/CD because we’re able to benefit more from automation since we’re using it in different stages before each release. If we’re continuously testing, then we should be shifting left and right at the same time, so these trends are fairly related to one another.
“If we are focused on continuously testing, we should be doing this in an agile way. Shifting left or testing in production, we need to be running automation in all stages.” –@fltoledo at @AbstractaUS during our @testim_io survey results webinar discussion. pic.twitter.com/VEBoc2hoFt— Tristan Lombard (@TristanLombard2) October 27, 2020
Automation Goals for the Future
“People do want to increase test coverage across browsers, but are also dealing with maintenance.” @shexman at @testim_io webinar discussion today. pic.twitter.com/Y4OlvaRCZM— Tristan Lombard (@TristanLombard2) October 27, 2020
Most respondents were looking to increase how many cross browser tests they run while many were satisfied with their current suite.
Oren commented that some of those who do not aim to grow their cross browser testing may be shooting for increasing their test coverage in a specific browser first. It may also be due to the high level of maintenance involved.
“…if approx. 90 percent of failures are related to something different than a bug, we should be focusing on making tests more stable. Yet, we want to automate more tests and run faster, more frequently….we need to [also] focus on having more reliable tests too.” – @fltoledo pic.twitter.com/gAaJ48gSuW— Tristan Lombard (@TristanLombard2) October 27, 2020
For teams that haven’t managed to already, it is clear that common goals include more automating of tests, integrating test automation with CI/CD, and increasing test coverage.
Federico commented that instead of making more tests, maybe it’s smarter to focus on making current tests more stable and more reliable.
What’s probably more interesting is where testers think the focus will shift in the next three years? From the results, it would seem that most expect to have already implemented CI/CD, with shifting left also losing importance as a priority. However, almost 80% answered AI-assisted testing will be a major trend looking ahead.
Discussing the @testim_io survey results today with @shexman and @fltoledo at @AbstractaUS. Do you agree with these results? pic.twitter.com/OIZqYNdBZK— Tristan Lombard (@TristanLombard2) October 27, 2020
COVID-19’s Impact on Test Automation
And last but not least, we couldn’t fully understand today’s state of test automation without mentioning COVID-19. The survey touched on how test automation has been impacted by the pandemic. According to the results, two-thirds (67%), responded that it did not impact their team.
Conversely, as for their budget, over three-fourths responded that it did cause their budgets to be reduced.
Oren speculated that the respondents must have had two different budgets in mind while going through the questions: one for people and the other for tooling. Teams may have invested in more tooling to compensate for not being able to hire more development talent during COVID-19 as their goals most likely still remained the same.
For us at Abstracta, one of the main impacts of COVID-19 was an acceleration in cultural shift. Nowadays, we see that our clients and other companies are more ready to accept the possibility that teams can work seamlessly in a distributed manner. As a benefit of this, companies can source the brightest talent, wherever it may be. We have been forced to learn more communication tools during the pandemic, which can help us to work in a more fluid, and even, asynchronous way.
“We have seen our customers creating more distributed teams around the world. Globalization changes the way that we hire and expenses can shift. You need to have communication skills and understand the tools for your team.”@fltoledo at @AbstractaUS during our @testim_io webinar pic.twitter.com/l9NApkfk6E— Tristan Lombard (@TristanLombard2) October 27, 2020
As for COVID-19’s impact on team’s test automation initiatives, 45% answered that they had been slowed down, 40%, accelerated, and 15% reported no impact.
What do you think of these survey results on the current state of test automation? Anything surprise you? Let us know in the comments!
P.S., Major kudos to Tristan, Testim’s Community Manager for live tweeting the webinar!
If your team needs some extra assistance this year to successfully automate tests, our remote automation engineers can help!
Recommended for You
16 Software Testing Podcasts to Listen to in 2020
Improve Your Software Testing Strategy: A Maturity Model
Quality Sense Podcast: Oren Rubin – Web Test Automation Challenges
Looking at some of the challenges of test automaton and how to pick the best tools and infrastructure In today’s Quality Sense episode, Federico Toledo sits down for a chat with the founder and CEO of Testim, Oren Rubin, an Israeli entrepreneur who has over…
Abstracta Opens Doors in Santiago, Chile, Launching First Franchise
(Spanish version below) SAN FRANCISCO, CA – June 27, 2019 – To meet the rising demand in global outsourced software testing services, Abstracta Inc. has opened its third Latin American office in Santiago, Chile. The newest expansion marks the inauguration of Abstracta’s first franchise, in…