Thinking critically about test automation from a developer’s perspective

In these two Quality Sense episodes, Federico delves into an entertaining and eye-opening discussion with Alan Richardson, a British consultant also known as “Evil Tester.” With more than 25 years of experience in testing and development, he offers consultancy and training in agile testing and test automation. Alan is the author of different books including “Java For Testers” and “Dear Evil Tester.”  He shares a plethora of content on his Youtube channel, podcast and blog.

What’s the Two-Part Interview About?

In this first half of the interview, the two covered:

• Learning test automation by automating games
• Ways to improve your test automation, like using JavaScript and modifying DOM elements
• Thinking tactically vs strategically
• The importance of critical thinking in test automation
• Using bias and skills of different individuals to form the best teams
• API vs GUI test automation

Continue listening in part two to hear about:

• Testability, automatability and observability
• How to show the ROI of automation, (Yes, you’ve gotta sell it!)
• How Alan got started in software testing
• What book he recommends on learning how to ask questions

Listen Here

Part 1

Part 2

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts
• Listen on Google Podcasts

Interview Transcript

Federico:

Hello, Alan. Nice to see you here and welcome to the show.

Alan:

Thank you, it’s just that no one else knows that we’re seeing each other, because you just release this as audio, so that’s that. I can’t show anything on screen. I have to describe everything I’m doing.

Federico:

I’m really happy to have you here on the show, and one of the things I wanted to mention to start is I think we’ve been interacting a couple of times on social media, but a couple of weeks ago I published an automation challenge on Twitter and you were one of the ones who answered the challenge, and I was impressed by the amount of people who coded for free or for fun. 

Like hey, this is my solution, using GUI and Selenium for trying to automate a specific game through a web interface. I wanted to see if someone could do it faster maybe. And you really hit me. Your solution was much faster.

I want to know, why did you engage in the challenge?

Alan:

The interesting thing for me is when I’m teaching testing, I like people to use real applications. To make that interesting, one of the types of real applications are games.

I write really bad games online in Javascript for people to test with. Part of that, I wanted to try to figure out how to automate it, and I’m interested in security testing and things like that.

So with these little games, I learned how to look at the DOM, look at the HTML, look at the Javascript that’s going on, and then start automating them from within the command line. Because I wanted to be able to write bots from within the browser that could play the games all the way through.

Games get harder and harder. So if you want to get up to level 160, that’s not what normal people do, so you have to think some way of automating that.

The easiest way of automating it is within the browser, where you have access to all the variables and memory. Because you know the positions of all the aliens, you can trigger the fire, and also you can bypass all the controls. You can make auto-fire work, because you remove all the limits, you can just change all the code and memory.

I was experimenting with this years ago, and then I saw an online talk of someone talking at a developer’s conference and he was doing this. He gave some examples of what he was doing, I thought that’s very similar to what I do, I went a little bit further, so I’ve just been pursuing that now. 

I think if we’re going to learn Javascript, it’s a nice easy language to learn, and you don’t need any tools whatsoever to learn it, because you already have it in the browser, and there’s the browser dev console.

You can do this on any site whatsoever, it’s just there to practice with. Once you know how to do it within the console, it’s a natural step to then, if you are doing any kind of GUI automation, inject Javascript into the page to support you with your normal everyday execution. There’s a side benefit that way.

If you’re not really doing automating but you’re more interested in interacting with it manually, you gain more technical knowledge of the application, and you no longer trust anything that the GUI presents to you, because you know how easy it is to manipulate and change. So there’s a massive amount of advantages and benefits like that.

The course that I put on Test Automation U explains a lot of the techniques and processes for doing that.

Federico:

Something that I think is very related when programming is that it’s a playful activity. Presenting things to do as a game or a challenge, it’s more engaging for people. I was amazed by the response of many people trying to automate in a faster way, in a better way, and I learned a lot from different solutions.

I didn’t see anything similar to what you did, and I think what you explained is the key. Not only using the record and playback tools you use when you are starting with Selenium or some tool like this, but also getting more involved with Javascript, with the DOM, all the technologies involved in the UI.

Alan:

For some people, that is an easy thing to do. We always want a mix of people. I say these things and people get nervous that they have to understand everything. But I do that because I’ve always been interested in the technology we’re working with, so for me that’s a natural extension, and I’ve always been interested in games. 

I wrote an interactive text adventure game, so there’s an online text adventure game that I use for training that can have 50 or 100 people using that. And that’s designed to help people use rest APIs, going to the DOM to see stuff that’s hidden.

Alan:

I think as soon as you give people games, they’re more likely to explore in more playful ways and not worry so much about the consequences. If you give them an application they stay very rigid, focused on the requirements. With games, they’re prepared to push the boundaries a little bit more.

Federico:

I have another example of something we were discussing yesterday. We were preparing our recruitment process and we have a challenge to present to people going through the process and it’s not the same to ask someone to automate. Like, “Access this e-commerce site, do a search, take all the titles of the elements that you find and save them in a CSV file” or something like this as asking them to “steal” the data of the database of this organization.

It’s the same thing, what you are going to do, but you present the problem in a different way, which is more engaging.

Alan:

And also people think about it a bit differently. One of the distinctions I make is the difference between tactical and strategic. If you give people a problem of “here’s an e-commerce site and we want to automate it,” they’re immediately going to think strategic. They’re going to think big tools to automate it long-term to really impress everyone.

If you talk about stealing stuff and scraping stuff, they instantly get tactical, and they want to suddenly look for a web driver, they might bring down J Soup, they might start processing the HTML directly, or use Curl. There’s a whole bunch of tactical, tool-based solutions. But if you ask them to automate an e-commerce site, they’re going to immediately get strategic, and start thinking about program code, web driver, page objects, and all the abstraction there, so it takes a little bit longer.

If you’re trying to focus on an objective, and you can give them the leeway on how to approach it, having the ability to do both is really important. That’s one of the reasons I do the Javascript stuff as well, so that I can tactically come onto a site and automate it without all the infrastructure.

Federico:

There was another game that you solved, I’m not going to mention which one specifically in order to keep the secret, but it was amazing how- do you know what you made me think about? 

About how critical thinking is really important also for automation. 

Because my challenge was “Hey, let’s try to solve this game,” and your solution was, “Hey, the solution is embedded in the HTML in some Javascript code on the client side. I don’t need to automate the user interaction with the game in order to solve it, I have the solution here.”

But I didn’t think of looking at the HTML or the Javascript in order to understand how the game worked, because if I had done that, I could have submitted much earlier.

Alan:

That’s one of the advantages I have, because I started as a programmer. I’ve been doing this for a long time, that I’m interested in how it works as well.

I want to make sure I’m automating from the right perspective, I want to make sure that I’m actually targeting the right things. A lot of times, we don’t understand the technology so we test the wrong things. I’ve seen teams come into a website, the little calendar drop-down controls, they start testing those. But those are the HTML5 built-in calendar controls; they’re testing at the wrong level because they don’t understand the technology. So I get in the habit of looking at the HTML, looking at the Javascript, looking at all the web traffic that’s coming through. It’s all part of my initial investigative scoping procedure, the reconnaissance I do on the site before I test it and automate it.

Federico:

There’s another thing related to the bias that we have as programmers or users of a specific tool. If we are in charge of automating a test case, or automating something, and I know Selenium or some tool, I probably will start thinking about how to solve it with the tool I know. But perhaps we can change the problem, the way we specify the problem, in order to allow them to think more tactically, as you say.

Alan:

That’s one of the reasons why I have that distinction between tactical and strategic. It gives me that ability to make decisions. The reason I developed that is because we used to do, we still do, agile. You’d come on a project that would be agile, so you’d do it the agile way, where you automate all the acceptance criteria, you work on it long term and build it up. 

A lot of agile projects don’t need to be agile. They’re not strategically agile. We’re building something in iterations that when it goes out there, it will only survive for a couple of months, it will never be maintained. We do not need a strategic approach to the development of that, which involves a lot of automated execution with Selenium and code bases and things like that. 

We could be doing it much more tactically to get equivalent results, to achieve what the customer wants.

“Having the ability to overcome our biases is really important. In order to do that, you have to recognize what your biases are.“

ALAN RICHARDSON

My biases are very technical, I’m oriented toward code, so I will look at those things on a first basis, which is handy, because I get a lot of information there. But there are some times when that is not appropriate. I shouldn’t necessarily look at the code if I’m doing accessibility testing or usability testing. There’s parts of the testing process that I don’t get involved in as much, because I like to harness what I’m good at.

That’s why you build teams. You build teams where some people are biased toward one thing, some people are biased towards another. As you approach your project more strategically, you mitigate the risk that only one person can do that by sharing the knowledge and skills.

On a tactical basis, if you can just harness people’s biases, you can be much more productive, and really quickly.

Federico:

I like this idea you mention of building a team with the different biases or different skills that complement each other with different visions. Maybe one has more visibility on the business or the code and can help you. If you are very focused on the code, maybe you need someone to tell you, “Hey, stop doing this, and pay attention to what’s important to the business.” Having this combination of things, it’s really important.

I was thinking of another thing related to bias. When I think about, “Let’s automate this,” typically most people, I would say, think automatically on the UI level. And maybe it’s not the best solution. Everyone knows about the test pyramid, the API level tests are easier to maintain, and maybe UI tests and GUI tests are a better solution for what you want to achieve. It depends on the context.

Alan:

It completely depends. You want to be able to maximize the opportunities that you have. A lot of the time why people focus on GUI automation is simply because that’s all they know how to do. Or simply focus on API and only know how to do API. So then, people use that bias to justify that type of automated approach. Whereas the more we scan the territory, the more options we have.

I was automating something last week that was all GUI based web driver, works fine on my machine. As soon as I put it on Travis in the cloud it was failing, so I converted it all to use J Soup, which I mentioned earlier, to hit the HTML, extract it, and do checks on the HTML code directly, because it was all HTML-based. But my initial bias was web driver because I was building it strategically and I knew how to use it long-term. As soon as I hit this environment constraint, that forced me to rethink whether I needed to use web driver at that point and I converted it to something else.

You really want the flexibility of being able to handle these things. Sometimes it is really important to automate the GUI, because that’s where the conditions can be tested. Unfortunately, when you want to automate that level, very often, the GUI needs a lot of support, because you can’t just automate a tiny bit. Sometimes you have to automate what flows through in order to get to that part. There’s a lot of work sometimes involved in that. But if you know how to mix, then perhaps you can come into the GUI, grab a cookie, throw it onto the HTTP stack, feed through some coils and come back to the GUI by injecting the cookie again. But you need to understand the whole technology process in order to do that.

Federico:

I think there are many people paying attention nowadays on how to reduce costs in software testing or in the whole development process but specifically beginning in software testing. Many people talk about using automation as a way to reduce or optimize costs of software testing. But I have another question. How can we reduce or optimize the costs associated with software testing automation? Do you have any insight about that?

Alan:

The easiest way, the instant way to do that is to not call it test automation. Spread the cost across the entire project. Because really what we’re trying to do is automate the assertions of particular conditions. Some of those relate to requirements, which span the entire software development process.

I tend not to use the phrase test automation, because I really want to get into people’s heads that what we’re doing is automating, which then frees you from a particular role… it frees you up from having to code it, you can use whatever tools are required, you can bring the team in, you have to start thinking about what are your objectives.

Because a lot of the time, unfortunately, teams themselves don’t deal in money terms. A lot of teams don’t have budgets, so they don’t think in terms of money. What they have is time, so what they’re really doing is spending time. Any return that they get is not going to be monetary-based, it’s going to be time-based.

“Most of the time when we automate, it’s to free us up to have time to do a different set of things that can add extra value. If we don’t automate well, then we don’t get that return, because we have to spend a lot of time maintaining, or fixing, or guiding through, massaging through, this whole automated execution.” 

ALAN RICHARDSON

But, it’s rarely about money, and it’s never about test automation. It’s always about automating to achieve certain goals. Some of those goals are related to testing, at which point it’s automating within our testing process, but it can still be done as part of the entire software development process, because testing is simply part of a software development process.

The more that we think in those terms, the more teams that will work together, will share skillsets, who will build things that are reviewed by the people who can review it.

Testers very often are not the best programmers in the world. Getting it reviewed by programmers should be a good thing to do. But sometimes programmers are not necessarily the best testers, so having them automate all the time may not achieve the abstraction layers that support testing to do ad hoc automation or to bolster their exploratory testing abilities. It all needs to work together as a whole.

Federico:

Again, it’s the team with the different skills complementing each other that is the key for success.

Alan:

I think so, and I think that’s what we have to fight for, working as a team all the time. Getting away from that role-based focus and making sure everyone is trying to achieve a certain outcome.

Federico:

Yes, Collaboration. 

Alan:

Very often people talk about testability, which I think is the wrong word, because I think the correct word is automatability. If we want to automate our applications, they need to be automatable. Because everything is testable. You can give me no instructions, get me in front of it, I will test it. It has some level of testability. But it’s mainly that relationship between me and the application.

When we’re automating, we want the relationship between the application and the tooling that we’re using to automate. We do have to put hooks in place. Well, we don’t have to put hooks in place, it’s just that there may be a danger later on for maintaining it. It may take longer to get up and running. It may take longer to do. So it depends on what we value. But if we can put ideas on things that are appropriate or use classes to help us identify them which remain stable, there’s really that stability between releases we’re looking for. And if they change, then it’s because they needed to change, which means we needed to change the structure of the abstraction layers.

Unfortunately, in those circumstances, it really helps if the people who are testing and automating understand the technicalities of what they’re working with, so that they know what to ask for. It helps to be really specific. It also helps if the people who are building the application experience some of the pain of automating it so that they can predict in advance what will be easy or hard to automate. 

Really what we’re talking about is how do you architect the application in order to be automatable? It is possible to build applications that you can only automate from the outside in because they haven’t got APIs in place. Or then if they do put APIs in place, it’s possible to double up the testing effort, because the GUI doesn’t use those APIs. So now you have to test from the GUI into the back end [inaudible 00:21:57] and you have to test the API. Whereas if the GUI used the APIs, you can do a lot of work by testing the APIs and then just do extra work to make sure that the GUI can communicate to those APIs.

What we’re really discussing at that point is architecting the application well to support automated execution and to support checking that it works at different technology tiers. 

We architect it in ways where we can check assertions at a certain point, then the risk and further up through the tiers is reduced, because it’s just a message passing up to them.

Federico:

There is another term that I have read many times recently, which I think is related to what you just mentioned, which is observability–

Alan:

I’ve used that term observability or observing for a long time when I’m talking about testing, just interacting with the application. I want to be able to observe the application, which means seeing stuff happen in passing, so I can see the messages. I then want to be able to interrogate those messages, to dive into them in more detail. I need tooling to do this. If I’m working on a web application, mostly what we get is the GUI. I have to then use tooling, like the browser dev tools, to observe the network traffic, to observe the DOM changes underneath, to observe how the CSS is changing, and then to interrogate after the fact.

Observability, I think, is adding tooling to hook into the application at different points to gather data that we’re not necessarily sure quite how we’re going to use, but we’ve architected it to open it as much as possible. There’s always the risk that we open it too far and then we’ve got security issues, but there’s always that balance between supporting people doing the work and supporting people who actually want to be malicious in your application.

Federico:

This is another different topic, maybe, but it’s also important in this goal of contributing to the test strategy or the quality strategy with the automation, which is how to show the value of what we are doing, let’s say, to managers or to product owners or to other people that should understand why it’s happening, how the investment in automation is paying off. 

Alan:

One of the things I try to do is avoid the obvious things that we’re asked for. Very often people want numbers or they want graphs. Very often they want those simply because they’ve had them before, and they really don’t understand what you’re doing, but if they get those they feel comfortable. 

What we’re really trying to do is we’re trying to create a custom process and communication flow for the environment we’re in. When you talk about value, value is a very unique thing to that individual. We have to ask them what it is that they value and it’s a sales process.

The way that I learned how to do this is not from IT books, it’s from psychotherapy books and sales books. I spent a lot of time using sales. There’s a bookcase behind me, and that’s pretty much sales books down to the bottom, sales and marketing books down here, cybernetics books and systems books at the top, but the rest of it is pretty much sales and marketing. That’s what we’re doing. 

When we talk about value we’re trying to sell what we are doing to someone else, so that they continue to fund it and continue to buy it. That means that we have to figure out what it is that they see benefit in, because you always present value in terms of benefits.

It’s not features, bug counts are features; benefits are “We have found all these problems before they have gone live.” If you really want to maximize the value in that, you try to explain how big they are, and how big a problem it would have been if someone else had found it after it was live.

When you’re talking in terms of benefits, it has to be the language of the person who is hearing it. Different people will hear it in different ways. Some people will hear the security value. Other people won’t care about that, because they don’t realize that that’s an issue. But if it’s a lost sale, then it’s an issue for them. So it’s phrasing it in the right ways.

I generally try to avoid numbers. Numbers and metrics are important when you are trying to change something, and you can objectively define that means.

You can say “we want to get faster at releasing,” then you can objectively measure your release process and use that as a measure to see whether the actions that you’ve taken are actually improving. But at a certain point, you drop that measure and metric, because it’s no longer of use to you because you’ve achieved what you want. If it’s easy to track, then you might add it into your process automatically to alert you if you start slipping. I generally try to avoid metrics on that basis, because we rely on them and they’re just meaningless.

Federico:

I think it’s more important to provide qualitative than quantitative information because you can misunderstand that.

Alan:

It’s certainly possible to read into numbers a lot, and for people to interpret them in ways you don’t expect. A lot of people talk about that storytelling nature of communication, that you have to tell this story around it. That’s because they are coming at it from a different perspective. Whereas I’ll look at it in terms of the sales and the values and the benefits, because that’s what I study. A lot of people study other things, so they’ll use different language to explain that.

Federico:

Something that I really like about what you said is that it’s evident even being a programmer focused on test automation or automating different parts of the process, there are many overall skills that are really important, related to communication to sell the value you are providing.

Alan:

Yeah, there’s a huge amount of skills. And they are skills that people don’t develop, like…

“The ability to install fear in people is a really important skill as part of that bug advocacy process. A lot of people try and explain how bad it’s going to be, but if you can sell fear in there, then people will help you, because that’s something we actively avoid.” 

ALAN RICHARDSON

There’s lots of techniques and approaches, which is why I study as much as humanly possible about communication.

You can do it in very subtle ways. If people tell you that something will not happen, you can be like, “That’s great. I’m so glad you’re sure about that.” You can be kind of positive, and they will interpret that in whatever ways: “You’re the one that’s supposed to be sure, if I’m sure then I’ve got blame, we need to really deal with this now.” There’s a whole bunch of subtleties around there in terms of how you communicate.

Federico:

I have some final questions for you. One is, I wonder how did you end up working in software testing or even, you provide a lot of trainings related to Selenium and automation, so how did you learn about that?

Alan:

I got into software testing because I was a programmer and I was working on a Jackson-structured programming diagramming tool, and I was writing a Kobal interpreter for it, so all very technical stuff, and that company went bankrupt, so then I got a job at another company writing test tools for them, then they went bankrupt, so since I’d just been doing testing.

Then I moved into a test consultancy and I was helping them build tooling to support them. Because it’s consulting, they put you out on site, so I had to learn testing as part of that. When it’s time I’m learning something, I learn it. So I read as much as humanly possible, I studied as much as humanly possible. And because it’s consultancy, you have to learn communication at the same time. So I’m pooling a whole bunch of things into this.

When it came to training, part of the reason I was writing books is because I was in a management level at that point and I was trying to recruit people and I did not think that the people I was trying to recruit evidenced the knowledge that I expected as a basic level.

So I wrote it down, these are the basic things, so there would be no excuse. I went through a period of time going, “We should have no excuses for not knowing this.” I tried to put as much information as possible to give people no excuses. And if they didn’t know it, then I could upsell them onto my book during the interview process and then maybe they could come back later on.

Also it’s a way of knowing that you’re going to have to train your staff.

When you’re interviewing someone, you see what gaps they have, and you have to be prepared to take on all the things that they can currently offer you and help them bridge the gap to get them to where they want.

If you don’t have the skills in training them, if you’re not prepared to invest the time in building them up, you’re really limiting the people you can hire and you may not be bringing in the flexibility you want in that environment, because you’re recruiting for a single skillset, whereas what we want, testing in particular requires a massive amount of skills. We have to interact with so many different people. If we don’t have the range of people and skills on our team or in our department, we are going to fall down. We’re going to rely on one person, and they’re not going to be there at that crucial moment. It was about bridging gaps between where people were and where I needed people to get to.

Federico:

It’s really true motivation, at least for me, to see people grow, to help people grow.

Alan:

Yeah, because you want people to fulfill their potential.

“People underestimate what they are capable of and they think they can’t achieve things, because they’ve got these biases and beliefs in their heads. If you can just push them forward, then they just get rolling with it. They surprise you then, because you did not know about all the other things that they have that they bring into play at that point.”

ALAN RICHARDSON

All the knowledge that they have from these disciplines that they’ve studied that you have no idea about, that they suddenly can relate to what they’re doing. It just boosts them even farther. That’s why I think the individual is really, really key, and really important.

Federico:

Something related to that about how to help people to grow as a tester, do you have any book recommendations for them?

Alan:

Read all my books, that’s a given. For me, a lot of the things that I have really relied on have not been related to testing and programming. That’s the easy stuff.

When I was taking consultancy and testing really seriously, I knew that I needed to understand how people communicate.

There’s a good book that’s called The Structure of Magic by Richard Bandler and John Grinder, it’s a two-volume book. And what they did is they looked at psychotherapists who were really effective, and who were getting work done really fast, who were changing people really quickly. They tried to identify what is the key to this, what do all these people do in common, what language patterns do they use, what questions do they ask, how do they know to ask those questions. That essentially is what psychotherapy, in particular brief therapy, is. You can make change with someone not after a year of therapy but in a couple of sessions and you do that through questioning. As testers, we ask questions. As consultants, we ask questions. As programmers, we are trying to build a model and ask questions as to what will best fill in, what won’t go wrong.

I just find all those skills related, so I recommend The Structure of Magic, it has some technical information on linguistics and how language works, but it’s really about how do we think about things and how do we understand other people’s models of the world and what questions do we ask in order to help them change.

Federico:

Interesting. I want to read it right now, right away. Is there something you would like to invite the listeners to do, like checking your YouTube channel, your podcast?

Alan:

Yes, I’ve got a YouTube channel, there’s podcasts, there’s books, and you can find it all at eviltester.com, that’s the easiest place. Everything in one place. But there’s books, podcasts, videos, blog posts, there’s too much stuff. And then you’ll find links there to Patreon, where I blog on a daily basis. I stick out a blog post every week, every couple of weeks, YouTube videos every week, every couple of weeks. 

What I really want to try and do is focus people’s attention on building their own models and owning their progress. On Patreon, I create a blog post every day to try to trigger people into thinking, “What did I learn today?” and to think about their approach to work, what didn’t go well, what are they going to change, what are they going to fix.

A lot of the time, I encourage people on a daily basis to write down what they do. At the start of the day, write down what you’re going to do, as you go through the day, write down what you’re doing, the links you encounter, the thoughts that you have, put timestamps in. At the end of the day, summarize it into lessons learned, because that helps you with your time management, your planning, but also with your exploratory testing, because we have to learn how to make notes. I make a daily note every day. I just have huge amounts of writing, which then improves your writing ability. Everything just spins off from writing things down, which is why there’s so much to find on eviltester.com.

Federico:

This is a great habit, a note. I typically do something like this myself. And I think this is also very useful for practicing gratitude and celebrating achievements, because it’s really important to reinforce positively what you have achieved and the efforts you put into everything you do.

Alan:

Yeah, I think so.

Federico:

I know that you don’t know who is going to be my next guest in the podcast, but I’d like to ask you to think what would you ask this person if you were going to interview him or her?

Alan:

What would I ask someone that I don’t know.

Federico:

Yes. I can tell you it is probably going to be a software tester.

Alan:

Now I’m on the hook, because I’m trying to think. I don’t tend to ask direct questions. What I have to start with is what do I want to learn, and what I want to know is, what is their secret for being as productive as possible and learning as much as possible?

Federico:

Excellent. I will ask this question and I think it’s going to be very interesting. Thank you so much, Alan, it was a pleasure for me to have you here on the show, and it was a lot of very interesting things we discussed.

Alan:

Yes, thanks so much, I can’t believe we’re finished already. The time has gone too fast. Of course, if I was listening on a podcast it might seem like ages, it might even be horrible.

Federico:

Thank you, Alan.

Alan:

Thank you! Bye.

---

Recommended for You

How Can You Optimize the Cost of Software Testing?
Quality Sense Podcast: Rob Sabourin – Testing Under Pressure (Part 1)