Detecting errors that automated functional tests often miss
Generally when we think of bugs in software, failures and unexpected behavior due to errors in the system logic come to mind. In part, by running (and automating) functional tests, we seek to detect these errors as early as possible. However, sometimes we miss other types of errors that are more obvious and recognizable to a user such as visual errors, which leads to the question: how important would it be to detect these errors within our tests?
In this post, Iâll explain how to achieve this with automated visual regression testing, advantages and disadvantages of this practice, and some tips to make it more worthwhile.
The Impact of Visual Errors on the User Experience
When talking about visual errors, we’re not referring to the aforementioned errors in the system logic, but to those aesthetic defects that cause interfaces to be displayed incorrectly, thus worsening the user experience.
Here is a clear example of a visual error in Amazon’s mobile experience:
The “Filter” bar is following us in this #GUIGoneWrong from @amazon! Visual #UI bugs sneak into production regularly, but Applitools Eyes is here to help đ ď¸ Reach out to learn how visual #AI #testautomation can keep your apps and sites visually flawless pic.twitter.com/IG4oVNC8iv
â Applitools (@Applitools) November 12, 2018
As you can see, the filter bar overlaps with the product description, which would be annoying for any shopper that wanted to read it easily. In this case, the error is an inconvenience, but there are other examples of when a visual error causes an application to be unusable. For example, when a visual component moves or disappears altogether.
Unfortunately, automated UI tests aren’t meant for visual validation, so theyâd miss this kind of error. As you can see by looking at the Amazon example, even tech giants arenât immune to these bugs.
To sum up thus far:
- As automated UI tests do not detect them, it’s possible for an application to have visual errors
- Itâs important to detect these bugs as they can be very detrimental to the usability and accessibility of an app, directly impacting the UX
Now that weâve covered what visual errors are and why they matter, letâs see how to detect them earlier in the development cycle.
Visual Regression Testing
Visual regression tests validate that the appearance of an application hasnât undergone any unexpected changes by first capturing a screenshot (a baseline) of the application and then, after each test, checking if it has changed with respect to it. If so, the test is marked as failed.
You can think of it like a game of âspot the differenceâ where you try to find differences between similar images:
Visual regression tests are especially useful for detecting whether visual errors were generated after making changes to the system. They can be integrated into existing automated tests frameworks (such as Selenium or WebdriverIO) using tools that allow visual regression. In addition, many of these tools allow you to apply various image comparison logics (for example, looking at the structure of an interface or the content), which allows flexibility in testing and obtaining deterministic results.
Should You Incorporate Automated Visual Regression Tests?
Now that we have a better understanding of what visual regression tests consist of, let’s compare the advantages and disadvantages of incorporating them into an automated testing process.
For more on test automation, read “When to Automate a Test.”
Advantages of Automatic Visual Regression Testing
- Increase the scope of your existing automated tests
- Detect visual errors early and quickly
Disadvantages of Automatic Visual Regression Testing
- Added maintenance costs due to the need to keep screenshots of web interfaces while taking into account their several variations such as on different browsers, devices, operating systems and more
- Image comparisons can return false positives leading to unreliable results and time wasted analyzing failures that donât correspond to errors
Taking these points into account, we can see that these tests help to solve the problem of detecting visual errors in a timely way. However, the high cost of implementation and maintenance that they imply can result in them being unprofitable. If that happens, a team could end up discarding them, so it’s key to implement them in such a way that these disadvantages can be mitigated as much as possible.
Good Practices for Automated Visual Regression Tests
- Target the application interfaces where visual regression testing would provide the most valuable results (where the highest priority or most critical cases are). This will help limit the number of images per case to focus only the most important ones.
- Always avoid making exact comparisons (pixel by pixel), since theyâre more prone to failure. Instead, use other types of comparison logic more suited to your needs (layout, content, etc.). This will help obtain more deterministic and reliable results.
- Consider baselines for devices where the application is most often used, and similarly, in their respective most used browsers. This will help to cover only the most important variants.
- In recent years, codeless testing tools are evolving to support the visual regression testing story. Most of them offer visual test recorders for end-to-end tests. Read more in the article The Case of Codeless Testing by Progress.Â
By implementing these good practices, itâs possible to make the most of the test results and reduce both maintenance costs and the probability of encountering false positives.
Visual Regression Tools
Last, but not least, letâs look at some of the tools that can allow you to incorporate these tests.
Applitools
Applitools is a visual testing platform that provides a library called Applitools Eyes available for Java, Javascript, Python, C #, Ruby and PHP. It can be integrated into automated tests at the UI level. Through this library, you can integrate various functionalities to your tests to visually validate your application. Its massive community and extensive library of learning material make this tool stand out, above all.
Check out this episode of the Quality Sense podcast, where our COO, Federico Toledo interviews Anand Bagmar, Quality Evangelist and Solution Architect at Applitools.
Percy
Percy by BrowserStack is a visual testing platform that, like Applitools, provides SDKs that can be integrated with UI-level tests in various languages. Its screenshot stabilization functionalities allow you to avoid false positives due to rendering of sources or animated images. For more information, check out Percy’s documentation.
Oculow
Oculow is an innovative visual testing tool that offers a web test management platform and a library available in various programming languages ââ(Java, Javascript and Python). Besides this, it offers a convenient functionality to manage baselines by applying artificial intelligence. Basically, it gives you the option to let Oculow decide for itself if there are visual bugs in the captured image or if the interface has changed. You can read more about it in the Oculow documentation.
Improve UX with Automated Visual Regression Testing
Ultimately, it’s clear that visual regression tests have a lot of potential since they allow us to test applications with a visual approach that we cannot achieve with other types of tests. In this way, we achieve a very important level of coverage against errors that we wouldn’t normally detect that directly affect the user experience. Incorporating these tests is a practice that I’d definitely recommend to any tester, especially because of the great importance that their results mean for the user experience.
Have you tried automated visual regression testing or plan to? Tell me about your experience in the comments!
Recommended for You
Quality Sense Podcast: Anand Bagmar – What You Should Know About Visual Testing
The Ultimate Guide to Continuous Testing
Tags In
Abstracta Team
Related Posts
[Ebook] A Complete Introduction to Functional Test Automation
Everything you need to know before getting started with functional test automation Introduction to Functional Test Automation It is often said that âAutomating chaos just gives faster chaos,â and not only faster, but also (paraphrasing a Daft Punk song) harder, faster, stronger… chaos. Yet, seemingly everyone…
How To Plan Test Case Automation with Development
A strategy to implement automated checks without losing sight of manual testing The goal of this post is to show a possible way you can plan test case automation with development in order to automate tests (have automatic checks) as soon as possible, without losing…
Leave a Reply Cancel reply
Search
Contents