Quality Sense Podcast: Bas Dijkstra – False Positives and Negatives in Test Automation

In this Quality Sense episode, Fede has a chat with Bas Dijkstra, a software test automation expert, consultant, and trainer from the Netherlands with over 14 years of experience in the field. Recently, Bas triggered an interesting conversation on Linkedin about software testers getting involved with unit testing, which sparked a variety of responses. Listen to the episode to hear more about that conversation, getting better results from test automation, and more!

Episode Highlights

• What should a tester’s involvement look like in unit testing?
• How to become a better test automation specialist
• Building reliable automated tests: how to avoid false positives and negatives?
• An important philosophy/book he recommends for being more successful and productive

Relevant Links:

• Bas’ Website: https://www.ontestautomation.com/
• Bas’ Linkedin: https://www.linkedin.com/in/basdijkstra/

Listen Here:

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts

Episode Transcript:

(Lightly edited for clarity.)

Federico:

Hello, Bas. So nice to have the opportunity to talk with you. How are you doing today?

Bas:

I’m doing fine Federico, thank you for having me. I’m excited to be on this podcast.

Federico:

Welcome. The first question I wanted to ask you is to understand that a little bit about your background. How did you end up working in software testing?

Bas:

Like so many people I’ve met and heard, it was sort of kind of by coincidence.

Federico:

By accident. Yes, I was expecting that.

Bas:

Yes, not really by accident, but by coincidence. Now I have a background in computer science. Funny thing is over here, the Netherlands I’m actually one of the exceptions. So I know a lot of people who are working in testing, at least over here in the Netherlands, and in Western Europe, that are wonderful testers, but don’t have a background in computer science. I am one of the few who does have such a background.

And after I finished my studies, I didn’t really want to become a software developer, but I wanted to work in software development. And I heard some good things about software testing, and about the company where I started my software testing career. And one thing basically led to another, and I started out as a junior software tester, and never really looked back.

Federico:

And I guess that this influenced the fact that today you are working in test automation, right?

Bas:

Yeah. Yeah, you’re absolutely right. So my background in computer science, where I learned a lot about software development as well, that starting out in test automation, that was a logical choice. I actually think within the first six months of my career as a software tester. So obviously I was still doing a lot of testing, but also I started exploring the area of test automation. This was back in 2006. So, and then slowly I started doing more in automation. First, just as a horrible test automation consultant, basically just creating very bad automation from way… regression test scripts that were way too long, just translating them into automation from the same perspective. And as we all know now, that’s probably not the best way to do it. But I didn’t really know any better back then.

So, but yeah, that’s what got me interested in test automation, and did a lot … I worked with … on projects for a lot of different clients. So I had the ability to see a lot of different tools, a lot of different domains. And yeah, that slowly I realized that this was really the area that I wanted to keep working in, and become more proficient at and yeah, start specializing into this specific area of software testing.

Federico:

And you mentioned something … The main topic I want to address today with you is related to test automation. But you just mentioned something that allows me to ask you one of the questions I have for you. Which is how to become a better tester in test automation. Because I read a LinkedIn post you wrote that I loved, where you mentioned that a way to become a better test automator is by learning how to better test software.

Bas:

Yes, yes.

Federico:

Can you elaborate a little bit more about that?

Bas:

Yeah. As I said, I made so many mistakes in the past with test automation, just by thinking that I could automate everything that the software tester did.

And I still see a lot of people thinking that that’s the end goal of test automation. Just to replace everything that a tester does, because it’s cheaper, it’s faster, it’s better, it’s everything. But yeah, as you know, if that’s what you expect of test automation, you’re going to be disappointed really, really, really quickly.

Why I think you should become a better software? Because to me, in my training courses, one of the first things I always ask, especially when I talk about introduction to test automation course, for example, is what are your views on test automation? And people always tell, “Automate existing regression tests.” Or, “We don’t need as much testers or testing anymore.”

And the definition I use nowadays for test automation is supporting software testing by performing specific tasks that a software tester performs, by the use of tools, just to make it more efficient. 

And that basically covers everything that why I think that you should become a better software tester first, just to get a better grasp of what it actually means to be a good software tester. And only then will you be able to see what are the things that a software tester does that can be done more efficiently by tools.

As you know, there are a lot of things that the software tester does that are just human cognitive tasks, that require creativity, and asking a lot of questions. Making judgements, and assessing risks. Tools are not very good at doing that. But there are also a lot of things that the software tester needs to do that can be done more efficiently by tools.

And that’s why I think you should become a better software tester first. Because test automation isn’t there to replace the tester, it’s to help a software tester. And in order to be able to better help you do better testing, and do parts of that software testing more efficiently, you should know better what it means actually, to be a good software tester, before you can apply tools to perform some of the tasks that a tester does, in a more efficient way.

Federico:

Yeah, and maybe differentiate which things can be automated and which can’t.

Bas:

Yeah, yeah, exactly. As I said, there are a lot of things that require a human brain in what a software tester does… there are unfortunately, there are still a lot of people who see a tester as only as someone who follows a script, and puts check marks in boxes. That’s not what a good software tester does. 

So yeah, it definitely is about understanding what it is that the software tester does, and then understanding which of the tasks of the entire spectrum of tasks and activities that encompass software testing, which of those activities can be supported by tools.

Federico:

And I guess that doing it right you can be more efficient, as you mentioned. And also gain more confidence on your processes, on your testing, maybe?

Bas:

Yes.

Federico:

But I’ve also seen that many people lose their trust in the test automation, in what they automated. Do you have any insights on how to trust your test automation, how to build it in a way you can really trust what you get from it?

Bas:

Yes. So as I said, I started out my career translating existing regression test scripts. Which were often multiple pages in length with… And they touched on maybe three or four or five different applications. Just automating them verbatim, just translating them one-on-one to automate the scripts. And automating everything from an end user perspective. So often using a graphical user interface, which as we all know, are the hardest automated tests to write, and everybody starts there. And because they are the hardest ones to write, there’s also a, and for those of you who have been working with tools like Selenium or Cypress, or any other tool that interacts with the graphical user interface of an application know, those are the hardest ones to get right. And with, “Get right,” I mean get stable, and present reliable information. Because for me, software testing is about uncovering information, and those tools and the test automation tools help you uncover parts of that information in a more efficient manner.

And so if you want to trust those tests more, you have to make sure that you can rely, safely rely on the information that those tests give you. And these tests, they can deceive you in a number of ways. And I often classify them, and I’m definitely not the only one here, in false positives and false negatives. And these are two different ways in which your test automation can be untrustworthy, so to say.

So let’s start with the false positives. Those are the tests that fail when there’s no problem in the application that they’re testing. So everybody knows about, and I’m not… I’m always using Selenium as an example, Selenium based tests. I have… I think Selenium is a wonderful tool. Really. It’s a great tool, and you can use it to write very valuable tests. The problem is, it’s really hard to make those tests valuable and reliable and trustworthy. So you should only have a couple of them, and make sure that they’re extremely reliable and trustworthy.

But unfortunately you see a lot of false positives there. So tests that fail, just because it took a couple of milliseconds longer than the maximum allowed wait time before a button appeared. Or because you had a different screen resolution there was this annoying popup that blocked the text field where you wanted to type some text. And I’m not going to go into the myriad of ways in which you can deal with this. But the problem with that is that people, and because these tests are so hard to write, people spend a lot of time writing that kind of test, and fixing them, and rerunning them. And say, “Oh, it failed. I’m just going to run the test again.”

I don’t think that’s a particularly good strategy to deal with those types of automated tests. Because how do you… If such a test fails often enough, people are going to say, “Oh, my test failed again. Okay. We know it’s unreliable, let’s just ignore it.” And that’s where things start to go downhill really quickly. Because if people don’t trust the outcome of your test automation anymore, what’s the value in creating them in the first place?

Federico:

If you were trying to optimize your testing, I’m sure you are not reaching that goal, right?

Bas:

No. So if all you’re trying to do is have fun with tools, and write code with tools, and that’s it, okay. But as you know, test, and especially test automation, it’s a means to an end. I want to uncover specific information that I can rely on, that I can trust, in a more efficient manner.

So, and these false positives are really not all too helpful in doing that. Because they create a lot of noise. And I’ve seen a lot of projects, and unfortunately I’ve worked in a lot of projects where this happened a lot, and people just ignored those tests. And the problem is if you ignore them long enough, people are just going to lose trust with the test automation efforts all together. And yeah. And just, “Yeah, we know we’ve got some tests for that. We built some tests for that. But they are not all that reliable, so let’s just have someone test those things again.” And so you’re going to do things twice.

And especially if you’re moving towards more rapid deployment, where you just can’t keep up with all of the releases that you’re doing every week, every day, every minute, I don’t know, you’re going to have to make some tough decisions. Are we … Yeah, because do we still want to test that? And if all you have is a bunch of unreliable tests, you’re never going to get to the frequency that you want to, in a safe and trustworthy manner.

Federico:

And what about the false negatives?

Bas:

What about false negatives? That’s a tough one, right? That’s definitely a tough one, yeah. Because false positives, they have one nice trait. They make themselves known. You’ll see when they’re there. Because your build is going to fail, your monitor on the wall’s going to flash red and says, “There’s a problem here.” Even though there might be a problem with your test, and not with the application that you’re testing. But at least they’re going to show themselves.

With false negatives, those are the tests that say, “I have passed,” but they didn’t detect a bug, or an anomaly, rather, because humans decide whether or not it’s a bug. They didn’t detect the deviation or the anomaly that they were supposed to detect.

And that’s a big problem, because those are just as untrustworthy as the false positives, but you don’t know, because they run… maybe you run them 10 times a day, 20 times a day. And they keep saying, they keep indicating, “I have passed. I have passed,” so everything is okay. But there still might be a problem, they just didn’t pick up on that problem. And those are the really tough ones to deal with, and to detect in the first place.

Federico:

And the problem is that you lose trust once you have a problem, probably in production with your users, with your customers. Because someone is going to say, “This is not working. This is causing problems.” And you see your test and you say, “I have a test that says that works.”

Bas:

Yes.

Federico:

“So I am not trusting you anymore.”

Bas:

Exactly. Because you will know that there’s something wrong in your software, it’s just not picked up by your testing, it’s picked up by your users. So, and then they’re going to lose trust. And as a result, the people in your team, or your manager, and then they’re going to say, “Didn’t we build an automated test for that?” Or say, “Why are we spending so much time and effort and money on creating those automated tests, and still things slip through?”

Now of course, you can’t just pick up or detect anything with automated tests. That’s just an illusion, because you can’t test for everything with automated tests, and a lot of types of bugs and defects that test automation just doesn’t pick up. But there is a category of defects that definitely should be picked up by test automation. And if you think you’ve got your test automation in place, but it just doesn’t pick up on the defects that it’s supposed to detect, yeah that’s a problem too.

So I’ve seen examples, and sadly, I’m not joking here, with code test code bases that boasted 95, or even a hundred percent code coverage, where there are a lot of tests that didn’t even check anything. They didn’t even have any assertions in them. They’re just … One of them actually included the comment, “It doesn’t have an assertion, but if it completes, then everything is fine.”

Federico:

No.

Bas:

Yes.

Federico:

You’re tricking yourself with that.

Bas:

Yes. Who are you kidding here? So those are the really difficult ones to detect in the first place. So, because they don’t make themselves known, but they can be just as, if not more, hurtful when it comes to trusting your test automation.

Federico:

The way I see it is like we are talking about trust. So it’s very similar to what happens with a relationship with someone. You don’t just build trust and forget it. You have to continue working on this relationship, on this trust, in order to keep it, right?

Bas:

Exactly.

Federico:

And really challenging.

Bas:

Yeah, exactly. So I think it’s very important. And trying to help clients and teams and companies that I’ve worked with just being aware of the fact that once you created a set of automated tests, you’re not finished with… You need to make sure periodically, and what periodically means is different for every team and every organization, that your test can still be trusted.

Yeah, you need to adjust test your tests when you’re creating them, but also periodically when they are part of your test automation suite. And just go through them on a regular basis and see, do they still have that defect detection power that they are supposed to have? Are they still valuable, in the first place? Because every test adds to your test code base, which means maintenance, which takes time and effort and therefore money.

So I think it’s really important. And I think there’s room for improvement with a lot of teams and organizations there, in just periodically going through your set of automated tests, and making sure that they can still fail, and they can still fail for the right reasons. Are they still picking up on the anomalies that they were supposed to pick up on?

Federico:

Yep, totally. The other question that I have for you, and it’s also because I read something really interesting in your LinkedIn timeline, it’s related to unit testing. Because I think you are the only tester I’ve seen supporting the idea that testers also have to contribute in unit testing. 

I think, I don’t remember someone else saying that we should take care of this as well. So what’s your experience? What’s your view on that?

Bas:

I think you’re referring to some thought I just… as you said, I posted on LinkedIn, like a couple of days ago. I was actually surprised by the amount of reactions that got, and the variety in all the comments. 

And I’m glad there are a lot of people who supported it. So I’m definitely not the only one. So I would never claim to be the only one who supports this notion. But there were also a lot of people who said, “I’m not going to do that. That’s the developer’s task.” And I thought, actually, that that was a little interesting; let’s call it interesting. 

Everybody’s entitled to their own opinion, but I think that as a tester, you … And I’ve never said that you have to. Some people thought I meant that testers should be responsible for writing the unit tests. That’s not what I meant to say there.

I’m also not saying that every tester has to do everything. Because some people said, “Yeah, we have so many other tasks, and how are we going to do this too?” I just said, “Just think about it. Talk about it with your team, with your developers, with other people, about what is it that you do in your unit testing? And how can I, as a tester, help improve your unit testing efforts?” Because I love unit tests, honestly, because they are perceived as difficult by a lot of testers. Because it involves code, and it involves knowing something about the internals of the application that you’re testing, about your application code base. Sure, that’s absolutely true. So there is this barrier there, maybe. But still, there are so many ways in which you can contribute as a tester, without being the sole person responsible for writing those unit tests.

Because I don’t think that’s a healthy situation either, that says, “As a tester, oh, just go and collaborate with the developer. Let’s just pick an hour somewhere and say, well, can you talk me through a couple of those unit tests that you write, and what do they do? What do they pick up on? What are they meant to test?” And maybe, if only to get a better understanding of what that developer’s actually testing with their unit tests.

And maybe you can already see well, if you’re testing that condition or that parameter value, why not test this one as well? Because I think that’s a possible… that’s something that’s a value that that parameter can take under these and these conditions. So maybe it’s a good idea to have a test for that as well. And only by that suggestion, you’re already improving the quality of your unit tests.

And the fun thing about unit testing is that they’re relatively, they run very fast. They’re relatively easy to write, once you know how to, and once you have some knowledge about the internals of your application code. As of course, given that your application code is structured in a way that makes it relatively easy to write unit tests. But that’s a whole different discussion that I do not want to go into, just testability of your code. 

Just by doing that,  you learn a lot about the system. You learn about what’s tested on a unit level, so you don’t have to repeat that on another level. And maybe you can do some suggestions, make some suggestions about how to improve that unit testing, those unit testing efforts. So, and because those tests run fast, they run often, they’re often run very early in the process, so you’re just moving a teeny tiny bit of your testing activities, a little to the left or to the front of your software development lifecycle, I think that can be very valuable.

Federico:

It’s very interesting. Because I am thinking now that doing those pair reviews or colleague reviews with a developer, you can learn a lot. And also instruct the developer about testing techniques. Maybe different ways to combine different variables in a more efficient way, using a pairwise coverage or something like this.

So you are contributing with your knowledge about software testing to the developer.

Bas:

Yes.

Federico:

And also it’s really interesting what you mentioned about learning what the developers are testing, so we don’t repeat the same things in other layers in the APIs, or in the UI automation, for example.

Bas:

Yeah. And I was honestly surprised by some people who were very vocal about, “That’s a developer’s job. That’s not something that testers do.” And I said, “Well, it’s called unit testing. Why shouldn’t you? Why are testers not allowed in to contribute to that?” And so it, and again, it’s not something that should be done just by a tester, it’s not just the tester’s responsibility, but I think that there’s a lot of room for improvements in a lot of development teams there.

Federico:

I have to recognize that I’ve been there. I’ve been saying that it’s not a tester’s task, but I do believe that we could contribute and collaborate with that.

Bas:

Yeah. So, it’s all a learning process. And maybe of course, in some situations with some teams, it works better than with other teams. And there are so many variables there. But I think why not just try it out, a big, a suitable piece of the code. Or maybe yeah, if you have a bug that showed up in production that you want to write tests for, together with the patch that you’re creating. Why not just pair with a developer on, okay, so what happened with this bug? Where in the code did it go wrong? What should we test for, to not make it go wrong again? And just start that discussion. And my experience with doing that has been very positive, because it does wonders for the collaboration between your developer and your tester.

Federico:

Totally. Bas, I have a couple of final questions for you. First of all, which book would you recommend people listening to the podcast?

Bas:

Oh, wow. I must admit, I didn’t give that much thought. And I always see also in my LinkedIn timeline, and on other social media and other stuff, I always see people recommending these self-improvement books, and business advice books and stuff. I’m an avid reader, but I read a lot of fiction. So at the moment I’m going through the entire bibliography of John Grisham. 

Okay, it’s not very intellectual. I use reading mainly just to wind down after a day’s work, stuff like that. So a general recommendation from me would be to just read more fiction, and don’t be afraid to let the people know, you don’t have to just boast, “See how many self-help and self-improvement and business books I put in my suitcase, when I went on holiday.” 

So, but if I’d have to recommend one nonfiction book, it would be Essentialism by, I don’t know if you’ve heard of it or read it, it’s written by Greg McKeown. And it’s, yeah, I’ve read it five or six times now. I try and read it every year, and re-read it every year.

And what it basically is about, it’s about doing less, and doing the things that you do in a better way. So I’ve got a problem. And my problem is that I have a hard time saying no, because I enjoy doing a lot of different things. As we already said in the introduction, I’m working on a lot of different things in parallel, stuff like that. And that’s how I like to work, but I need to, sometimes I need to refocus again on what’s really important, and what’s maybe less important. Or what can be done at a later time, or what doesn’t contribute to where I want to end up in… I don’t know how long. So this is a book that really helps me there really. Because it gives you a lot of practical tips about figuring out what’s really important, and learning to say no to things that aren’t as important.

Federico:

Thank you for this recommendation, because now I think this is what I need to read, you know? It’s a skill I need to develop!

Bas:

Yeah. I’m the same. And I’m sure there are a lot of others who struggle with this as well.

Federico:

For sure. Okay. And the last question is, would you like to invite the audience to do anything?

Bas:

If you want to react, or have some opinions on what we’ve discussed, I’m always happy to talk with fellow testers, and people from development, and basically anybody who’s involved in software development, about test automation.

So just send me an email, or send an invite on LinkedIn, I’m pretty active over there as well. And if you’re at all interested in training in the test automation space, that’s, as we discussed before we started, that’s one of the things that I mainly do these days. Yeah, just contact me and see where I can help. My trading portfolio is online at my website, and I’m pretty sure you’re going to include that in the show notes. I’m not going to mention it, because just rattling off a URL just doesn’t really work.

Federico:

I will include that for sure. And consider that now it’s easier to participate in your workshops and trainings because you’re offering it online, right?

Bas:

Yes. Because of the whole COVID situation, a lot of the training that was supposed to happen live and in person in the classroom or an office somewhere, have moved online. So I’ve gained a lot of experience over the last six months or so with conducting training online. So, yeah. And that also opened up my training services to teams in companies who are in different timezones. Yeah, absolutely.

Federico:

I’m sure that we all agree that it would be better to travel to the Netherlands, to have a face-to-face training with you. This would be a much better option, but nowadays we have to deal with this.

Bas:

Yes. I’m looking, and things are slowly opening up again, at least over here. So I’m really happy to slowly start doing face-to-face training again in the next couple of weeks. But yeah, the whole travel thing, which I like to do as well a couple of times a year, that’ll probably not happen for a while. So yeah. And then doing training online is a great alternative.

Federico:

Yeah. Great. Thank you so much, Bas. I really enjoy talking with you and I learned a lot.

Bas:

Thank you so much for having me, Federico. It was a pleasure.

Federico:

Have fun. Bye-bye.

Did you enjoy this episode of Quality Sense? Explore similar episodes here!

Recommended for You

How to Avoid False Positives and False Negatives in Test Automation
Quality Sense Podcast: Michael Bolton – Software Testing & Automation