Quality Sense Podcast: Anand Bagmar – What You Should Know About Visual Testing

In this Quality Sense episode, host, Federico Toledo interviews Anand Bagmar, a professional from India with over 20 years of experience in the software industry. Today, he’s a Quality Evangelist and Solution Architect at Applitools. In this episode, the two uncover the basics of visual testing and how it can enhance a team’s test automation efforts. Anand also describes the challenges associated with visual testing and how Applitools helps to overcome them. If you’re involved with or interested in test automation, you’ll take away some interesting ideas from the conversation, including the importance of following a value-based and risk-based approach.

Episode Highlights:

• What is visual testing and what are its different use cases?
• How to think about visual testing as a part of one’s overall testing and test automation strategy with the aim of maximizing ROI
• Automated visual test maintenance: How to create baselines and deal with changes therein
• Resources to become a better tester, Anand’s productivity tips, and more

Relevant Links:

• Follow Anand on Twitter – https://twitter.com/BagmarAnand
• Get a free trial of Applitools: applitools.com/
• Access courses to improve your test automation skills: testautomationu.applitools.com/

Listen Here:

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts

Episode Transcript:

Federico:

Hello Anand. So nice to have you here on the show. Thank you for accepting the invitation and welcom! How are you doing?

Anand:

Thank you so much, Fede. It’s a pleasure being here. I know we’ve been planning this for some time. I’m really glad we could finally make it happen. Yeah.

Federico:

Finally! My first question for you, how did you end up working in software testing?

Anand:

Well, the answer is actually quite similar to what I guess a lot of people would have experienced how they got into software testing and truth be told, it was not by choice as a post option for me. 

Like most other people, I was fascinated by code. I wanted to get a job in development after I did my Masters in Chicago in computer science. And I said, okay, based on a good opportunity that came across, I got a chance to move to the Bay area where all the magic of where all the action is, but the opportunity I got over there was to start off in testing and then move on to the development road. And that’s how it started. I said, okay, it’s a stepping stone. I’d start there. I’d learn a few things, I’d understand the industry, the domain and everything and then I can move on. 

But, I got hooked into the types of things I was doing over there. I started with WebMD. That was my first job and started off with testing and did some performance testing as well. Did some interesting IVR – interactive voice response based testing as well. And unfortunately that’s when the recession hit, so that’s the reason they got shut down. 

And then it was a struggle finding a job, the next job over there. And of course, being on a work visa, it was even more challenging during those times. Then I got a job, the only thing that I could get was a job in customer support at Borland Software for their Visibroker Corba product. And I said, okay, something is better than nothing at least it could start paying the bills. So I started there, but that clearly to me was a completely transformational period for me, the kind of policies and procedures that we had.

And I’m not saying that was bad in that sense, but it was appalling to be on the customer side, who is facing the issue. 

And it sort of became a mission for me. What can I do to help build a quality product inside out without having the users of the software have to go through this kind of process crap that might be dead, right?

So this happened in 2000 to 2002. So all of this time, since then I’ve played various different roles, any role that you can think of, most likely I’ve touched upon if not made it significantly as part of various teams, but the whole focus has been, how can I do this to the best level possible to prevent issues coming up in production that will impact the users? Of course it can never be perfect, but how can I play my part in doing that? So that work experience at Borland Software, that to me, was the beginning of my career in testing. It was not WebMD necessarily, but it was Borland Software.

Federico:

Wow. That’s very interesting because yeah, it’s, it’s true that most of the people I asked this question, they say it was by accident or by chance, I was looking for something else. And finally, I decided to give it a chance and I liked it. But you also found special value in it. So you, you understood in your first experience how the testing has, has a special value and that’s maybe different from most of the stories I’ve heard.

Anand:

Thank you.

Federico:

Interesting. So let’s talk about our main topic today, which is visual testing. So for starters, can you give me an introduction? What is it, why should we pay attention to, to this? What type of bias or errors can we find or prevent with this type of solution?

Anand:

The name itself is pretty indicative about what it does. It’s visually testing what is coming up on the screen or in the app for your product. Now, technology has evolved. Processes, practices have evolved. And with the collaborative way of working, fast feedback approach of working, we have gotten very good or pretty good in terms of using software tools, technology to try to validate if the quality of the product is to the level where it needs to be using automation. And this automation could be at any level, right? 

From unit testing, API testing, contract testing, workflow testing, UI testing, and also we’ve got the NFS or security performance. So many other types of things that can be done. DevOps plays a huge role in making sure the same builder is propagated correctly, automatically, configurations allow you to just point to different environments. 

So we’ve gotten pretty good in terms of being able to test functionality from a tech perspective, as well as a business value perspective and CI helps us run this on every change that happens. We’re building confidence in a much better way than ever before. But of course you cannot disregard the aspect of the value that humans bring to the process. 

Unfortunately, what happens is, instead of having humans test what cannot be automated, test what else that could be there that’s interesting from an end user perspective, unfortunately, a lot of times these people spend time just looking to see if the screen is showing up correctly, but just testing what has not been automated, or if the automated tests are failing for whatever reasons, trying to reproduce those issues and spend time on that. 

So what happens is they end up spending less and less time in exploring the product further on what is not automated to find out more interesting issues potentially that exist in the product. And they’re focused on these mundane activities, which code should have caught easily. Also as a result, what happens in this, because now there’s an explosion of the form factors, what the end users access your product on, right?

It could be on the left laptops or desktops wearing screen sizes, resolutions, or on the mobile devices of varying different form factors, portrait landscape, more, that also becomes a different way. So what ends up happening is the cursory look at the screens in terms of validating the functionality you end up missing the obvious things like are images even showing up correctly? Is there overlapping content or not? Has the layout changed? Is it working in the correct responsive fashion based on what I’m seeing the page or the app on? Right. So there are a lot of visual issues which get missed out. And in fact, a lot of functional issues also could be existing which the tests have not been implemented for.

If I’ve got a long page, which I need to scroll to see the lower contents, but I’m not going to be able to see that long page for each and every screen or browser or device to make sure it is rendered correctly. What if that is a problem in the backend code or in the UI code where a certain functionality is not being seen correctly, etcetera, that’s a functional issue, more than a visual issue, right? But it is manifesting as a visual issue as well. You would very easily miss out on that by looking at the standard automation tools that you have, or even with the human looking at it, because it’s just not feasible to look at all these combinations in a fast paced release environment. 

Visual testing comes in and adds a lot of value by allowing you to use a combination of the existing automation tools and combine aspects of visual testing with them to get the results back that will let you know if functionally and visually, everything seems fine and in all of the supported browsers, view port sizes, and mobile devices as well.

And now with this combination approach of functional and visual with the differences or results that are being seen, then the humans can again apply that mindset and see these differences that are reported, are they really valid or not? Right? So that’s where the combination of functional and visual becomes really powerful to see how we can leverage and increase the use of technology to reduce the manual effort, (that’s error-prone) required to find these out. So that’s what the visual testing area is, right? 

What is rendered on the screen? It could be a pure visual not functioning, but how can you get that very quickly?

Federico:

It’s not only about the aesthetics of the web pages or it’s like giving more power to your existing automation in order to validate so many more things with the same tests.

Anand:

Absolutely, absolutely. And there are a couple of aspects to it again, right. It’s not just about a static page. It could be a dynamic content. Today it is Anand logging into the website. Do I see the name Anand over there? Tomorrow it could be Frederico or logging in over there and he’s got maybe a shorter name or a longer name. Does the rendering change in the UI based on what the text is displayed or is the name getting trimmed down, for example, right. If it’s longer than expected, is it getting trimmed down and can the test really capture that? Cool. So dynamic content also becomes a very interesting aspect from a visual perspective.

Federico:

So I guess this is one of the challenges when you are implementing the visual visual test automations and what are the challenges you will face if you try to implement these tests?

Anand:

Well, challenges are plenty, and they will be operated as different forms for that matter. One challenge is about, it’s a typical way how you would get started with implementation of any tool or technology, right? You read up about it. You will see some samples around it. And most of the samples are people generic and make the concept understood to the readers. 

We start off with the “hello world” type of example. Okay. If I have to start with a new programming language and see how to write a simple hello world first, and then I’ll progress into implementing some more features, functionally based on what I really want.

So the challenges are two-fold, right? How can I think about visual testing as a part of my testing strategy and as part of my test automation strategy, making sure I have the right environments, the right type of data and based on what type of control I have on these aspects, am I using the right approach for visual testing? Right? So that is the first challenge. 

The second challenge of course, is the kind of issues the product itself can throw out to you based on different ways it can be interacted with or worked on. All right. So again, it can depend on what if I’m in a low network or a poor network condition? Is my page loading correctly or not? That’s going to give me false positives in terms of differences found from a visual testing perspective. What if the data is changing so much that it just doesn’t make sense to compare this data now from a visual perspective, because the results are not going to be accurate any way, right? There’s just too much dynamic nature happening over there for the comparison to happen.

The two main types of challenges of visual testing are not having the right testing, test automation and visual test strategy. So just using those technologies to do any random thing. And second, of course, is not understanding your product well enough to see if visual testing is applicable and if it can actually help you make the right type of decisions based on the results seen.

Federico:

This is very connected with the following question I wanted to ask, which is if there is a specific context or a specific type of application that you say, okay, this is a no-brainer. I must add visual validations to my tests because I’m going to get the best ROI from here.

Anand:

Yeah. So it all comes down again – sorry – it’s a long answer again, because it’s not very straight forward to really say this is when it would work. I do a lot of blogging as well. At least I used to, I do this on Blog spot. Hopefully the site can produce WordPress on anything. You can host your own content. What is the risk of the blog post not being seen correctly in different browsers, on different view port sizes, etc? Minimal.

Okay. But what if this was a product website, a marketing website, what is the response, certain rendering issues or visual issues happening on that product? Because people are actually going to be doing research based on that and they’ll be purchasing products potentially based on that. Right? So the risk becomes a very important aspect to think about. Then, this is a no-brainer if I should use it or not.

So I could choose to use visual testing for my blog, but it’s not going to make much difference. But, what is a no-brainer is to see what type of issues potentially have leaked in the past, or what is the risk of certain issues happening in the product and what is the impact going to be on the users? 

So, one example, which I often like to use, right? If I go to the Sony or Bose website, typically if I want to buy a high definition TV, I’ll go to the product company’s website and do research on that. And then I’ll also compare with various other sites to compare the features, see the reviews. 

And then personally, what would be your impression if the Sony website was a very poorly rendered website because of whatever issues? What do you think about spending $5,000 on an ultra high definition if Sony cannot even build a web page correctly? That is the risk. That’s the impact of it. Brand reputation, revenue loss, though Sony is not selling it directly from their website necessarily, but it’s an impact in the sales that happen. 

So the risk is a very big factor to consider: What is the impact if I do not do visual testing here? I am doing all other types of testing, but do I have enough eyes on the ground to look at the product manually every time before the release happens, to make sure everything is fine, or can I again leverage some better technology or tools to help give that feedback to me in a much quicker fashion?

Federico:

Now, going in the opposite direction. Can you tell me which things would drive you to decide not to include visual testing in your test strategy?

Anand:

Okay. First thing is the risk. What’s the risk of things going wrong? And if the risk and the impact of that risk materializing are not too high, it’s not too bad, it’s okay not to do it. 

Same thing for automation as well, you should do it only if it’s really going to add value to you in the long run. So the value and the risk is going to be the main thing. If it’s not going to add too much value, then why should I want to increase another aspect in my tech stack? Because every new line of code that you write, or every new tool technology that you incorporate, is a tech debt. You have to maintain it. You have to continue evolving it to make sure it continues adding value. So if it’s not required, if it’s not going to provide value, don’t use it.

If the kind of visual differences that you see are not going to be valuable to you to take meaningful decisions. So for example, this is a problem that we see a lot in, based on the type of visual testing tool that we use, right? 

If it’s a static page, then almost any tool can potentially work. Again, if you’re doing an apples to apples comparison, because most of the tools use pixel based comparison. So if I have to open up a webpage, my blog webpage, let’s just stick with that on the same machine, in the same browser at the same screen size of viewport size every time, then I can use pixel magic.

But if my requirement is I want to see this page on all the different browsers at different viewport sizes, then I have to think about how to create my baselines. How do I evolve my baselines for new blog posts or whatever changes I do? And at different screen sizes, do I have the bandwidth and the appetite to create all these baselines and maintain it as well to do the correct comparison? Because the rendering is going to be different. The visual testing will not add value if I’m trying to compare an apple with an orange. Okay.

So that’s the kind of thing that I would say if this is my use case, it’s all static. Visual testing can help again, the risk value is an aspect to it, but if it’s completely dynamic, for example, if I take the Amazon homepage, for example, right? Or you could take any e-commerce webpage, there’s so much of a dynamic nature that happens over there, you really have to think if visual testing will add value over here? 

And I’m not saying it doesn’t, but if on every second day, you are going to change the style of the page… Say the banners are going to be small or large or if there will be new offers or a new promotion, which means the layout keeps changing very dynamically, then maybe that is not the right place to do visual testing on that nature of a dynamic page. If you have control over the data to make sure the data changes what I want to see that are five layouts possible. And I want to check each layout of those five, from a visual testing perspective, then it will add value.

So there’s a lot of contexts that you’ll need to think about to see if it really is going to add value or not. And then accordingly go ahead with it.

Federico:

Well how about the maintenance of all of these, because you’re mentioning a combination you have to maintain the baseline images of. And I can imagine that if you start to combine different things, multiple test cases, different screen sizes. So probably if you add more automation with visual variations, you have to take more care of more content, right?

Anand:

Absolutely. It’s a nightmare to maintain because if you think about the evolution of visual testing, the way it started, right? Visual testing is easy in that sense. What does it really mean? I have something as a baseline. So if… let’s say I go to a homepage or I launch an app, the first landing page of the app, I will have a screenshot of that. 

Every new build of that particular product, I want to take the same screenshot again and compare it with the baseline to see if it is the same. Now, the traditional open source tools or the traditional tools for that matter for visual testing, the way they do this comparison is going to happen like this: They’ll take a screenshot and with simple code, you can say, compare this image to the other. There are a lot of ways you can do that comparison, but essentially it is doing a pixel to pixel comparison, right?

So now here’s the first problem that happens. What if you want to run the same test on a different device? A different device may have a slight change in the screen size, or the resolution might be different, or you might open it up in portrait mode versus landscape mode. But think about it, the screenshot that is captured, even for the same portrait to portrait comparison, if the screen size is different, is that picture really the same? When you’re doing a pixel comparison, it is not going to be the same, right? 

And that’s the problem when it happens, you run automation, not just on one device or one browser, not just on one viewport size of the browser. You want to see if your app is responsive, which means there are so many different combinations. So if you say the popular browsers, Chrome, Firefox, and Safari, right?

And if you take three standard viewport sizes in terms of testing your responsive nature, then it’s four multiplied by three images. That gives you 12 combinations just dealing with web where you need to see if your product is rendering correctly and is seen correctly. That means there are 12 baselines that you would have for each version of your app!

Federico:

For a single page.

Anand:

For a single page, right? For a single page, a very important point. And now with a new build, your product is always a living, breathing product, right? That’s why you create a new build. There’s something changing in that product. As a design changes, or as you evolve the functionality of the product, the baselines that you had earlier are useless because now your product has evolved. You now need to add new versions of that baseline and test that single page to make sure the comparison is going to happen correctly.

And that is a problem about how do you create those baselines? How do you evolve those baselines? And of course is there a comparison happening to the level that you really want? Is it doing pixel based comparison? And then again, a challenge with pixel based comparison to overcome those challenges about those small variations, they started introducing a concept of tolerance, right? But why is a 5% tolerance limit of differences acceptable and 5.1% is not? 

There is no science behind it. It is just taking a number from thin air and thinking, is this really good to add value to me or not? Maybe this 5% tolerance is enough for me to say, and if that is acceptable to you, then it’s absolutely fine. Okay. But what we see is you will end up missing a lot of real issues with that tolerance, and you will still end up getting a lot of false positives because of the tolerance that you set.

So there has to be a better way to do that comparison. So the main challenges when it comes to visual testing is of course the accuracy of the comparison. How can you eliminate the false positives and the false negatives, right? It’s how do you create those baselines effectively in a very seamless fashion? And more importantly, how do you update or evolve those baselines as your product functionality evolves? You don’t want to be spending your full time just doing that. It defeats the purpose of automation.

Federico:

Yeah, for sure. Totally. Well, how can I get started adding visual validation to my tests? What’s the best way to start?

Anand:

So there are many ways to start off. There are a lot of tools in the market that you can use that are open source and can get started with to understand visual testing. 

The recommendation that I can give is go to applitools.com, sign up for a free trial account and you get almost all the functionality of Applitools that you’d be able to use as part of your automation or just, there are various different ways you can use Applitools for visual testing. 

The key part about the key challenges that Applitools solves are the three challenges I mentioned about visual testing, right? The comparison itself, Applitools uses AI technology. And this is not just a buzzword that is being used. Applitools, in the platform, has got more than 1 billion tests executed on the platform over time. There are more than 400 enterprise customers across all the domains who are using this to get value out of it.

So the AI algorithms are really powerful and you can choose selectively based on the context of your tests, based on the context of your application, which AI algorithm is right to use for that particular screen or comparison. Right? So it solves the challenges of pixel based comparison.

Federico:

So it helps in defining when a change is something that you introduced intentionally or when it’s a potential bug. Right?

Anand:

Absolutely. Right. So I’ll give you an example. If it’s a static page that I’m loading, I’m just loading it in various different browsers and viewport sizes, and it’s a static page, right? So you can use a strict guard, which highlights the differences as seem to the human eyes. So if I’m loading the same page, in the same browser, same viewport size for every new bill, I want to see even the smallest difference that a human eye can capture as a difference… whether it’s a font change or color change that is there.

It will tell me what that really is. However, if the data is not really in my control and there’s dynamic data, then I could use a layout guard where you focus on the structure and integrity of the pages instead of the content of the page. Okay? 

So if there’s a lot of dynamic content that is coming up, let’s say a stock price app, right? Or a finance-based app, but the content might keep changing. The stock of course, the prices might keep changing, but you want to make sure the layout and everything else is working fine. You could use a layout at the moment there for such pages. So you have a choice from a lot of which to use and you can use a selectivity on portions of the screen as well. That will give you the right type of value for the type of test that you are implementing.

So remember, it’s all about the visual testing strategy that I spoke about, right? You have to choose what to get from them. What portions of screens you want to test that are going to give you value to reduce the risk, right? You can experiment with this using your free Applitools trial account.

In fact, another great feature that happens also, I guess, is the Dropbox cloud where you don’t have to worry about cross browser testing, necessarily. You can use our test cloud to say, I’m going to run my tests just once, locally, but Applitools will take care of making sure the rendering and validation is happening across all the supported browsers and viewport sizes and mobile devices automatically based on the configuration that I’m interested in. So you don’t have to have that infrastructure on your site. You can just use our Dropbox cloud to do this scaling across browsers and devices and get the validations as part of a single test execution.

So it’s a very powerful way to get fast feedback and scale across the platforms. But this still means we have only solved one aspect of the challenge in terms of the accuracy of comparison, but what about the other two aspects as well about creating the baselines and updating the baselines?

There’s been a lot of effort put into the designing of the Applitools dashboard and the beta implementation happens that Applitools takes care of managing the baselines for you. You don’t have to worry about, I have to take 12 baselines for that one screen on my own. Applitools will take care of that for you. As you see the differences with every new build that comes in, though Applitools uses AI to figure out the differences and very accurately. It is 99.9% accurate on algorithms. The differences shown are to the team member who can actually make decisions on those differences because a team member knows if the product has evolved in terms of functionality, but is this a regression that has been introduced?

So Applitools does not aim to solve the problems which are out of its control. You know, what has changed in the build as expected changes or you know what the differences shown are regressions or not. And then you will make decisions on those. And if the product has one, it’s a single clear, simple click of a button that you can update the baselines automatically. And the next time that test, if we compare it to the new baselines. So it’s a simple click versus managing all those baselines on your own on this, and then updating it into some data store of stuff. So these are the three challenges that Applitools solves and you can easily get started experimenting with these by using a free trial account. You can sign up on applitools.com

Federico:

Great. I will share the link in the episode notes now that we are already inviting people to doing things, maybe we can also invite them to this Automation University.

Anand:

Right? So yes, sign up for a free account and get started. There’s a lot of good documentation over there, but a very important initiative from the Applitools perspective is a community initiative called Test Automation University, where we not only have three courses related to visual testing that are, there is a lot of content, which is a very well curated and created by experts in the field across all different aspects of test automatization programming to using various different types of tools from automation perspective. 

I would strongly encourage everyone to keep that passion for learning and curiosity and use this great resource, which is available for free. You can take these courses at your own pace, and they’re also post learning parts based on what your interests might be, where we will suggest a different set of courses, what you can take to help you in your learning journey. So please do take a look at Test Automation University, for that purpose.

Federico:

Yeah. I can tell you from experience that we consider this an amazing resource for our testers in Abstracta. I have to thank you and your team for providing this amazing, amazing resource!

Anand:

It is great. And I’ve taken a lot of courses myself because, hey, I don’t know everything! I’ve been in the field, my hair is gray, I’ve been in the field for more than 20 years, but there’s so much more that I still need to learn. Yeah.

Federico:

That’s amazing. Last question for you. I really believe that in order to improve our productivity, we have to pay attention to those small things that we do everyday, multiple times. So I’m curious about what, if you have any trick, any habits that you can suggest people to form or to avoid?

Anand:

So there are a couple of things that I try to do sincerely for the value that they have been bringing me. One is for the lack of a better word, I have a big to-do list. And this list is constantly evolving based on what new things that I want to learn or experiment or try out. It could be ideas that I want to implement or blogs that I want to write, or new tools, technologies, processes, books that I want to read. This list is ever evolving. And do I get to all of the items in the list? Definitely not. Do some items never bubble to the top of the list? Definitely. Yes, but it’s very important to have this guiding list behind the scenes and very actively keep taking a look at it to see, is there something that is a low hanging fruit, or that is very crucial to me for the next set of challenges that I’m going to encounter.

If I do this now, it will help me. And that list constantly keeps changing. So as part of this conversation, it reminded me that I have a couple of pending courses on Test Automation University, it’s on my list. I need to bubble that up and do that, right? So I constantly take a look at this. This is a very good way for me to keep my passion of learning and curiosity going up as I read things as I come across, things mentioned by others and added to the list, and I’ll try to explore that further and it’s okay if I don’t complete it, but at least I’ve given it a shot and see if it’s adding value to me or not. Right? 

So keeping learning and having a way to track the different things you want to do is a very important thing.

The second thing, which has helped me a lot, especially in these trying times, is pretty good. It has been a very weird year due to the COVID-19 virus. I’ve been working from home continuously in that sense. And it’s not really easy to just be sitting in front of a computer every time. So invest in exercise, take frequent breaks, drink a lot of water. I went back and invested in a standing desk. So that is helping me be more energetic and keep more focused as well. So that is also something that I would encourage others to think about. What can help them to be healthy and be at peace of mind as well. So that’s-

Federico:

That’s really important I think. Very, very insightful thoughts on that and thank you for sharing them and thank you for all of this interview because I really enjoyed it. It was really amazing to learn from you and your experience. Something that really resonates with me is that you pay a lot of attention to the risks and the value of prioritizing efforts, according to that. So thank you. Thank you so much for the conversation and all the insights.

Anand:

Thank you. It’s been a pleasure. 

Did you enjoy this episode of Quality Sense? Explore similar episodes here!

Recommended for You

Quality Sense Podcast – Bas Dijkstra – False Positives and Negatives in Test Automation
Webinar Recap: The State of Test Automation in 2020 – Testim Survey Results