A look into software testing as a social science as Michael tells the story of how he became a tester

Carrying on the work of some of his personal mentors and the great thinkers in software like Cem Kaner and Jerry Weinberg is no easy task! But, that is what software testing consultant and trainer, Michael Bolton has set out to do. With James Bach, he’s co-authored the Rapid Software Testing classes and methodology. Over the course of his career, Michael has traveled over a million miles to over 30 countries around the world as a keynote speaker and instructor.

Having met before at the testing conference, TestingUy, Federico invites Michael onto the Quality Sense podcast to discuss his unique approach to testing and how he arrived there.

What to Expect?

• Michael explains how he accidentally became a tester… (don’t we all?)
• Many amazing metaphors that explain the purpose and often the problems with today’s software testing
• Why it’s mistaken to think that tools can eliminate the work of a software tester
• The two important kinds of testing and why you need both to make sure you’re creating the software you’ve intended to

Listen Here:

• Listen on Soundcloud
• Listen on Spotify
• Listen on Apple Podcasts

Episode Transcript:

(Lightly edited for clarity.)

Federico:

Hello, Michael. Thank you so much for accepting the invitation of participating in this podcast. For me, it’s an honor to have you here.

Michael:

Oh, the honor is mine. Thank you.

Federico:

You know, I want to start talking about board games, because I love board games and I know you like them, and this is related to what I consider the biggest achievement of my life! 

Because I remember a couple of years ago, we spent some nights hanging out in Montevideo and I had a chance to play set game with you, which by the way, is an excellent game to practice pattern recognition and things like this. And I could win a couple of hands, do you remember that time in Montevideo?

Michael:

I do. I have a lot of experience playing set, and I’m not a great player, really. I don’t think of myself as a good player. You get better with practice, right? To some degree. But when it comes to playing set, I don’t even have a look in on Paul Holland, for example. I don’t know if you know Paul.

Federico:

Yeah, I met him, but I didn’t know that he was good at it.

Michael:

Oh, he’s insanely good! It’s ridiculous. Paul’s good at a lot of things, a whole lot of things, but he’s just a vicious set player.

Federico:

I was impressed by how Paul can manage the interaction of a group of people with a K-card. I think.

Michael:

Yeah. The K-Cards, the facilitation thing. Yes. Different kinds of cards, I guess.

Federico:

Yeah.

Michael:

He’s good at all kinds of things associated with cards, he’s been a magic tricks too, which is another thing to do with cards. So there you go.

Federico:

Interesting. Michael, my first question for you, this is something that I don’t know about you, how did you end up in software testing?

Michael:

As so many people do, it was a crooked path that starts with being born as a human being. 

I just had this happen to be sitting at my desk here, this little tape measure, and little kids, get little kids, a tape measure and they play with it. And they pull it out and they look at the marks on it and then they do one of these? And they snap themselves inside.

So we had these experiences of learning stuff and observing stuff and interacting with stuff and experiencing painless stuff sometimes too. So we’re all born to be testers. And I was brought up in a family. My parents were, my mom was an English teacher. My father was an Anglican priest, but they both valued learning and education and so on. And they instilled that in me. 

I read a lot and learned a lot, but the first time I, well…another way, another path, another step in the path of being a tester was I got a home computer somewhere around 1986 maybe. It must’ve been earlier than that, ’84, ’85, somewhere in there, I got an Atari 800 computer.

And I was curious as to how the thing worked. And it was relatively inexpensive to find out about that. So I became fascinated by the internals of the thing without ever actually being good at it. I was very curious about how these things work. So I remember buying a bunch of books, “Mapping the Atari”, was one, right, “Mapping the Atari Computer”, that talked about the registers in the processor and memory locations and all that sort of stuff. And I didn’t really grasp it terribly well at the time, but I was fascinated by it.

My girlfriend at the time bought an IBM PC, well a clone and a laser printer because she had the idea of a typing business, a transcription business. And so figuring out how that worked, learning how the machinery worked, and once again, I kept buying books about how the guts of it worked and how to take best advantage of it, and the tips, tricks, and traps books and the Peter Norton books on the internals of the machine and so on.

And that sort of stuff is just always fascinating to me. So when it came time to, I was working in theater at the time. So this was definitely a hobbyist thing, when it came time to take some time off in theater, because theater work is inconsistent, I started working for a headhunter and it’s a long story that I’ll make shorter. 

I started doing data entry for them because they needed to locate resumes and CVs for the applicants, for the candidates. And they had a huge backlog of it. And they said, well, our problem is we can’t keep anybody on the job to do this. And I said, well I’ll happily do that so you can find my CV more easily. But I realized that I could apply tools to this job.

At that time, this is like 1988 or so, people didn’t have a concept that there were, not only could you use a computer, but you could put extra stuff onto the computer to help you do stuff on the computer. Now, you know what I mean? It sounds like a commonplace to us, but that was the state of the world back then. So I snuck in from my personal collection of software, I put some stuff on the company’s machinery so that I could do my work faster, more easily.

And then I realized that the application that they were using kept its data in a form, such that I could write code to get to that data in a way that was more efficient than the application did it. So I’ve been doing the same kind of thing lately with my manager. It’s all just XML. Well, in this case, it was all just the DBF database format, right? So I could write programs. 

And that is when I really started to think about testing because the people who I was writing the software for were not experts. They wanted to get stuff done. And as a programmer, I worked, which is a title I kind of just stole. I grabbed the job. I invented the job for myself in this organization. Oh, you’ve got a programmer now!

So I could not have faith that people would be able to run my stuff without problems. So I had to put myself in a position where I would try to encounter the problems myself. And that, I think it’s very shortly after you become a programmer that you realize you’re a tester as well, if you’re good at programming. 

So that was, it was sort of interesting because I was not coming at programming from the perspective of being a programmer. I was not a computer science student, right? I had not been born into that or not gone through an education for that sort of stuff. And what was important to me was how people got to use the product. 

Now, for a lot of programmers, when they go through computer science school, the kind of training to get in how to test stuff, doesn’t involve interacting with the product and experiencing it in the same kind of way that my flavor of testing is all about.

They’re much more into formally proving algorithms. And if they’re getting any kind of education in testing at all, it’s usually at the unit level, right? Showing that these inputs map to these outputs and they do so in an appropriate way, but they don’t…it’s a general pattern so far as I can tell in computer science school that they’re not focusing computer science students on general systems. They’re not focusing computer science on user experience. You get more of that in computer engineering schools, I think. I suspect, rather less than computer science because that’s a branch of the math department.

So my flavor of testing is very much grounded in the idea that I was close to the people who are using the product. And I was aware of their desire to get stuff done without friction, without problems. 

And so that’s the kind of testing that I started to focus on. Then I moved to the Canadian office of a company that was headquartered in California. I eventually ended up working down there as a customer support person, technical support. 

And in the technical support role, once again, you’re exposed to the fact that there’s a wonderful thing that David Platt said in his book, “Why Software Sucks…”, he says, “People don’t want to use software. They don’t want to use your software. They want to have used your software.” Right? The software for them is a means to an end. They just want to get something done. 

People don’t want to drive somewhere, they want to be somewhere. Relatively few people enjoy the process of driving particularly much. They drive because they want to be someplace else. But then Platt also pointed out something really interesting, I’m fast forwarding to 2008 when he gave this talk, he said, “This is why geeks, programmers…” He did this very close to you, you’re in Berkeley, right? So he did this at a conference in San Jose.

He did a poll, he asked, “How many people here…”, (audience are programmers, mostly a few testers, few program managers, and other software people, mostly programmers) so he says, “How many people here drive cars with a manual transmission, with a stick shift?” And, way more than half the hands go up. 

Now this is in America where about 12% at that time, that somewhere between eight to 12% of the population were driving cars, with a manual transmission, and this was over half, way over half. And he points out the geeks, people who are involved in technology like stick shifts, they like manual transmission because they enjoy the process of driving. They like that sense of control over the car.

But most people don’t want that. Most people just want to get from somewhere to somewhere else so they can get the advantages, the benefits of being someplace else. And they want it to be as friction free as possible. So my users were like that and I was aware of it because they were right next to me.

Federico:

You know what? This is one of the things that I had to adapt in the way I used to drive because in Uruguay we have manual transmission cars there.

Michael:

How God intended cars to be.

Federico:

And now that you mentioned that, I miss it a lot. Driving with a manual transmission, I feel more control over what the car is doing, even though what I want is to be somewhere when I drive.

Michael:

Right. But you’re engaged with it.

Now I don’t want to make too much of a parallel to this because I hate the notion of automated testing and manual testing. What I do like though about that metaphor about that parallel is a sense of engagement, right? 

You feel more at one with the car. You are not an object at that point. You’re not a package that is being transported from one place to another. You’re engaged with it. You’re part of the thing. You are part of the system of the car and getting you there. 

And it keeps me awake and it keeps me alert and it helps me to notice things and helps me to pay attention to the state of the car. It’s really interesting because sometimes I notice people who are…

I’m in their car as a passenger, I’m saying, “Hey, do you hear that?”, something about the engine or something about the car and they go, “What?” And the funny thing about it is it’s never occurred to them to pay attention to the car that you’re trying to make you look the other way. 

So there’s a kind of paradox there because I want my…the users of my software when I’m making software, I want them to have a friction-free kind of experience. But at the same time, I’m a little worried about it because I do want them to be engaged with the product, to the degree that if there’s something that the product is not doing, just the way they want it to do. I want them to be aware of that. 

And I almost kind of want them to get frustrated when there’s a problem, so they can bring that problem to my attention so I can help fix it for them.

So it’s kind of weird. I’m on both sides of the fence when it comes to that, but it’s absolutely the case that I, through my career as a developer and as a tech support person, I just, I kept seeing people in pain because of decisions that I’ve made as a programmer or things that I’d sort of forgotten, just swept under the rug as a programmer or about things that our products, when I was doing tech support for Quarterdeck or in the system around our products was making stuff harder for people, in a way that I deemed kind of, it doesn’t have to be like this, things can be different.

So somewhere along that journey, I met Cem Kaner. I had become a program manager by that point, but the head of the testing group at Quarterdeck had was bringing Cem in to teach a class to her testers. And she knew that I’d been fascinated by his stuff. 

In the middle period, between the time I was a tech support person, and the time I was a program manager, I was a tester, I worked really closely with testers and its program manager. I was engaging with the testers all the time. 

I identified as a tester, right? I’m still a program manager. And so I met Cem and it was Cem who really got me excited about testing. His style of teaching and his style of talking about testing as something far more than poking the product a little bit to make sure that it didn’t fall over. 

Cem’s style of testing was about a deep study of stuff. It was about teaching testing is not simply this technological thing, but really as a social science. 

A few years later (I guess it was 2003 or 2004), I invited Cem up to Toronto to give a talk and to teach some classes and stuff. And his talk was titled, “Software Testing as a Social Science”, where he was taking this really serious approach to the ideas that they have, social science ideas, rather than physical science ideas, with respect to testing.

The thing that I remember most clearly from that talk was the idea that testing software doesn’t give us perfect answers about how the software is doing. It gives us partial answers that might be useful just like social scientists do.

It’s not like the physical sciences where you have a very straightforward means of analyzing things. You can formalize those things. You can use very specific, formalized measurements for things that appear on some kind of scale, some kind of physical scale, but with software, we’re really examining how people feel about stuff.

And we’re examining relationships between people.

Federico:

You’re not defining rules that happen all the time in different circumstances.

Michael:

Yeah, that’s right.

Federico:

It’s precise.

Michael:

Right. So it’s got to be heuristically based, right? It’s got to be not, it can’t be algorithmically based, strictly speaking, and to the degree that we have algorithms and software, the algorithms allow us to follow a recipe. Given this set of inputs, we get this set of outputs and algorithms are, as it were infallible, right? 

Algorithms don’t fail. That’s the essence of it. You put this in, you get that out and it terminates, that’s the other thing, right? An algorithm ends. But the algorithms aren’t solving the social problem as such because the social problem still exists. And here’s an example of that, which James Bach appointed me to a really good example.

Long division is an algorithmic process, right? Dividing. So we go out for a meal, you and I, and one other person, we’ll invite Claudia, our mutual friend. And so you and I and Claudia go out for dinner and the bill comes back and it’s $150 US, very nice meal and we had a nice bottle of wine. Of course, how do we divide the bill fairly? Well, one way we can do it is we could say 150, let’s see one, five, doesn’t go into one, but one does go to 15. 

So we divide it by three and there we go, okay. Three, sorry, five, three, doesn’t go into one. So 15, five, and then zero. And that, okay, good. $50 each, except you only had one glass of wine and Claudia and I split the rest of the bottle. Plus I had the appetizer and the dessert and Claudia had the dessert. And all you had, was you just had the main course and that was that. 

So is this fair or mathematically it’s fair, but socially it’s not, right?

Federico:

The equation is going to be more complicated than that.

Michael:

Yeah, that’s right. And it won’t be an equation either. It’ll be some kind of weird system diagram or something, but of course we sort that out socially and I say, okay, well, listen, I should be putting in 75 bucks. And Claudia had the dessert. She didn’t have the appetizers and I drank the majority of the wine. So maybe actually it should be 85 bucks. That’s a much more complicated thing to resolve. 

But there’s a problem in the way we think about software testing. And that is what we do is we put in 150 and three and we get 50 out and we say, everything’s okay. 

So we have to ask ourselves, is the software solving the problem? The mathematical, the functional problem, the capability issue, is it doing the calculations properly. Well, that’s important, but we also have to ask ourselves a question, is the software helping us to solve the social problems? 

Is it helping us to address the job that people need to get done? Well, that’s been an evolution all the way along through my career in technology generally, but in testing especially, and I put down to that Cem Kaner who introduced me to James in 2001, and James said, get your ass to Jerry Weinberg’s community.

Federico:

Great advice!

Michael:

Yeah, really. Jerry was the master then. So Cem and Jerry, between them and James, and I sort of immodestly say I, attempting to follow in their footsteps. Jerry’s shoes are awfully big so it needs a lot of people to fill them. 

Fulfilling the humanist notions around testing has been what my kind of evolution has all been about and all the little things, how I got into it have been founded on trying to scratch that itch of the fact that software isn’t solving problems for people as well as it might.

Federico:

It was one of the hardest programs, because once you, it’s like defining the equation that you want to solve, because once you have it defined with mathematics, it’s very straightforward to find the solution.

Michael:

It is.

Federico:

But the problem is understanding which is the equation you need to solve.

Michael:

Well, that is an important part of the engineering process, right? Not just as, did we get the right answer, but have we picked the right approach to get the right answer. But then the right answer is not always what we want or at least a certain kind of answer is not always what we want. 

We keep running into this problem. It seems to me, I’m about to do a thing and attempted to do it really want to do it, wherein I test, it’s based on experience a couple of weeks ago, I test this idea: Can you print an envelope in Microsoft Word easily? 

Right? Now, that’s not a matter of, well, I did Sir, I asked for 14 points, Futura font on the letter. Do I get a 14 point Futura font? That’s not the deal for me. Although it’s an important part of it at some point, but what I want to know is, can I get that job done?

And in my experience last week, I don’t know if anybody saw the tweet blast about it. It took me WAY longer than half an hour. And it was SUPER irritating all the way along.

Federico:

I can imagine.

Michael:

It was confusing and it was difficult and it was surprising and it was annoying. And that is something that we’re going to miss if we treat software testing as a normal technical problem, as a mathematical problem, as an algorithmic problem, not to say that, that stuff isn’t important. 

And I think James and I are starting to frame some language about the way we could look at two very different flavors of testing. One of them is a kind of testing that programmers do, the programmers should do, programmers need to do as they’re building something, focused on what we would call the discipline frame, right?

We’ve framed this  in terms of discipline. And we don’t want to make it sound like a punishment, that kind of discipline. We want to make it sound like this is what good engineers do. This is what good craftsmen do, they have a certain kind of discipline about their work. 

And in that case, the kind of testing that we’re going to do is going to be testing that is focused on a correct output. It’s going to be focused on, it’s going to tend to be focused on the unit level. It’s going to tend to be focused on things that are machine checkable. It’s going to tend to be focused on automated checks. It will also include review and pairing that kind of testing, with the intention of answering this question: “Are we building the thing we think we’re building? Are we building the thing that we intended to build?”

So if we’re thinking in terms of bridge-building tactics, right? Are the nuts and bolts in the right place, right? Or has it been riveted properly and are the materials, have we done a tensile strength test on the materials and so on.

Federico:

It’s like we understood the problem and it’s like, checking that we are solving the problem. We understood the problem the user has.

Michael:

Well, that’s the other kind of testing it seems to me. That’s the other kind of tests, not the discipline kind of testing, but two other forms of testing.

… two other frames for testing. One, the intention frame is that do we actually know what the user’s problem is and how we intend to solve it? So we think of that in terms of intention, and then what we’re coming to call the realization frame, which is a pun. 

Realization having two meanings. One, have we realized our goal, have we achieved our goal? Have we brought this product into reality in some sense? Is it a real product now? That sense of realization, but also the sense of realization that goes like this, “Oh-oh, I just realized there’s a problem.” Realization in the sense of recognition. 

Recognizing that there might be a problem here. Now that is a different flavor of testing. It’s still uses tools, we can still use tools to help us do this, but the mission of this kind of testing is to ask ourselves, are there, no matter how disciplined we’ve been, no matter how well we frame their intentions and no matter how well we prepared for this moment, are there still problems that have eluded us so far that we’ve not quite identified? And that’s the kind of testing that we get with experience with the product.

So in my bridge metaphor that’s the kind of testing that we would ask, “All right, what happens after this bridge is up there in a high wind?” Remember those pictures of that bridge in Seattle/ Tacoma or the London Millennium Bridge. The other thing where they said, “All right, let’s see what happens when we actually put a bunch of people on it.” And in both of those cases, in one case the bridge shook itself apart, in the other case, very carefully re-engineer the bridge so it could handle that kind of structure. I’ve walked across that bridge. It’s sort of cool. That’s the kind of testing that requires experience with the product.

Federico:

I remember there is a funny bridge in, I think it’s in some city in Spain, in the North, that it’s made with glass, I think. It’s really beautiful. It looks amazing. It’s like a piece of art, but when it’s raining, which it typically rains in the North of Spain, it was very sloppy, so it was very hard to walk through the bridge. So they had to put some extra layers of some other material.

Michael:

Right, right, right. That’s the thing about engineering is we’re not aware of these problems until we’ve gained some experience with them. Now that speaks, it seems to me, to an idea which is not honored very much in this business, the idea of examining history, examining the history of technology and examining the history of science as well to see how wonderful ideas that we’ve had along the way haven’t always turned out exactly as we thought or hoped that they might. And we’re very bad at that, in the technology business it seems to me partly because it’s a young industry and it’s fueled to a great degree by the energy of younger people.

Certainly that was the case when I was in a Silicon-like Valley company in the 1990s. We were all young. We were all ambitious. We weren’t tremendous students of history to the degree that we could have been. 

I think that’s why to this day, we have people coming along and selling software, trying to sell software, trying to design software where they say, “Hey, we’ve got a program. We got this idea for a program that will check other programs. And all you have to do is feed program A into our program, we’ll call it… Our program is program X because that’s cool. Program A, the program that we want to tap, we’re going to feed that into program X and then program X will tell us about all the problems in program A.

Well, even the most modest student of computing history knows about the halting problem. And a more diligent student of the history of technology will recognize that as early as the 1950s, certainly in the 1960s, people were coming along and saying, “Hey, we’ve got this fantastic idea for a program, a program that can tell whether another program has problems and it can find all the problems in this other program.” 

So it used to be, I don’t know what they used to call it, but now they’re calling it AI. And they’ve got these machines that are coming along and saying, “Hey, look, we don’t need actual testers. We’ve got machinery that can tell you whether there’s problems in your product.” Well, to some degree, that’s true, but it’s been true as long as there’ve been compiler checkers.

Federico:

You mentioned something before, an example before with driving cars with the shifts, with that automation it’s like you forget about the curiosity. You don’t pay attention to maybe some weird sound in the engine or things like that. So probably this is one of the risks of just trying to automate everything and say, “From now on, this system is going to take care of everything for me.”

Michael:

Well, let me tell you about a company that I went to in Denmark not too long ago. What they’re doing is really fascinating. What they do is they make really, really sophisticated audio instruments. These fantastically sensitive microphones that they train on jet engines to notice changes in the sound and the operational profile, and to put those changes in front of a human being to interpret. 

So that’s the kind of tooling that I’m fascinated by, not tooling that replaces testers in any sense, or not tooling that purports to find problems, but tooling that amplifies our ability to find problems. 

And if we thought about testing tools in that way, I think our testing tools will be a whole bunch better. We would have more modest aspirations for what the tools can do, but we would also place them in their proper context and their proper perspective, the things that tools can do and the things that people can do.

So for example, these tools helped designers of cars, and they showed me how it worked, to give a sense of what it would sound like in the car for the rider and how they could make little adjustments to the car to dampen the sound of the engine and to dampen the sound of all the other machinery that’s ran in the car, which is not insubstantial by the way, and create a more pleasurable experience for the rider by doing very sophisticated analysis of sound in the car. So these are tools that help with analysis, but they don’t for a moment suggest that we’re going to replace either mechanical engineers or audio engineers in the design of the car, it’s just going to make those people super, super powerful and that’s fantastic. That’s great.

Same thing, I was about to talk about compiler checkers. Compiler checkers help programmers save a whole ton of time by looking at the program lexically and syntactically and saying, “Are there any obvious, easy errors here where the programmer has expressed yourself in a way that will cause memory leaks or that will cause the product to fall off at the end of an array or something like that?” 

Those kinds of tools are straightforward, not too hard to write, have gained increasing sophistication over the years especially as languages have gained flexibility. There’s room for interpretation. So now the compilers will flag something, “You sure you really meant to see that?” Without actually saying that there’s an error. That sort of stuff, that kind of assistance for programmers in IDEs and things like that has been enormous. It’s been really, really amazing. It has made programmers a lot more productive.

Testers I don’t think we’ve done exactly the same thing. What we’ve done for testers, we’ve done something really dumb, I think, which we focused our attention in testing tools on saying, “All right, well, let’s see, what did we have before?” We had a bunch of procedural, scripted test cases that would try to turn testers into machines saying, “Do this, do that, do this, do that, type this here, click that there.” And then see if you get the correct output at this point, based on this thing we already figured out. 

Well, that was a pretty dumb idea to start with for testing, but what makes it even dumber is to take a human and try to turn that human into machine. It’s even dumber to take the process of trying to take a human, turn it into a machine, and then mechanizing that process so that what we’re getting is a machine acting like a human that’s being asked to act like a machine to try to find human problems in the product! It’s a crazy way of going about it!

Instead, let’s go to the equivalent of compiler checking for the discipline frame. Let’s put that in the discipline frame, give that stuff to the programmers, help the programmers out to some degree in constructing ideas about risk and ideas about possible problems, and maybe even a little bit of coding and application of those tools as testers let’s do that. 

But as testers, let’s put our attention into the realization frame where tools can help us, but really what we’re trying to get our minds wrapped around is the experience of the product. 

And in those things, we’re not only looking… I’m not just talking about user experience stuff, I’m also talking about the kind of experience we might have we, using a bot, allowed the machine to do a whole ton of very, very simple and straightforward transactions to see how it responded under load, to put big loads on the product and to do it that way.

Not to bother so much about simulating what the users do, we can do that kind of testing at the API level pretty straightforwardly and pretty easily, but here’s something that we can’t do very well which is to have the machinery follow a task, give the machinery a task to do and tell the machinery, “Go do this task and tell me about any problems that you run into when you do it.” 

Well, the machinery won’t do that. Machinery is not good at doing that, but if we give to the tester this assignment, here’s a job that somebody who uses this product is trying to get done, try to get that job done. And here are some ways, first of all, just try to do that and then report on any problems you find with that. That’s a good start.

Second is, all right, here’s some variation you can put into this, not just the way we intend for the person to use a product, but here’s some ways in which people (we’ve noticed patterns of this because we’re social scientists and ethnographers and we go out and look at this, people that’ll actually use software) and we come back and we say, “Here’s some things that we’ve noticed people doing.” 

There’s reasonably foreseeable ways in which they could misuse the product or use the product in ways that we hadn’t intended, or that we hadn’t really thought of very much earlier on. Have you ever seen a use case that includes a step five and then the user closes the laptop lid?

Federico:

What happens then?

Michael:

And again, well, you never see that in the use case. Everybody in the use case, everybody is always very old-trained. They’re always prepared to do the job. They’re never distracted. They never have to reach for a pen or a pair of scissors. The boss is never tapping on their shoulder and saying, “When are you going to get this thing done?” Their kid or their cat is never interrupting them because they’re working from home. That’s never there in the use case. 

So as testers, I think our evolving role is to start to bring out that aspect of testing, the experiential notion of testing from the user’s experience perspective, but also to think about what the ops person’s experience with the product might be like. 

If a number of users are seeing cascading failures, how’s the ops person going to respond when their job is to maintain the back end and to identify, “Oh my God, the database is getting out of sync with what people are actually doing?”

I remember back in, it was in 2000, it was just after my daughter was born, 2004, there was a big problem at a big bank in Canada because a programmer put through a one-line change in their product and it was a big batch process, it starts up around midnight and about three o’clock in the morning, the company that was responsible for a certain kind of transaction process and related to that batch process said, “Hey, wait a sec, wait a sec. Something’s off here.” And they reported back to the Ops people at the bank, the Ops people said, “Oh my God, okay, we gotta get this fixed. Let’s back the change out.”

Well, that’s really straight forward. From the programmer’s perspective, “Oh, yeah. I’ll just go back to GIT or  whatever and bring out the next to last version and fix the problem. Okay. And now we can put it back.” Well, there’s another issue and that is that there’s an entire three hours where it’s a batch database transactions, which now have to be rolled back. 

So from the programmer’s perspective, fixing the problem took a minute or something really easy. From the perspective of rolling back the database, freezing the database and rolling it back and restoring all those transactions and restarting them, that took two weeks. So it seems to me that there was a step missing somewhere in the bank’s process, which is let’s get a little bit of experience with this one line change on a simulated system. One that provides us with a reasonably realistic set of experiences to go through and let’s see how that goes. And let’s alert ourselves to the possibility of those kinds of problems. So I’m not just talking about user experience, I’m also talking about organizational experience, operations experience, systems experience with things as they’re changing and as are being built.

Federico:

There’s a huge backstage behind a system, right? There are so many pieces and all of them play a role in the final user experience, right?

Michael:

Absolutely, yeah.

Federico:

I guess we should pay attention to all of them. Related to what you were saying about AI and automation, I remember a quote from Cem Kaner, which says that automation tools should be a way of expanding the reach of the testers not a way of that way of replacing them.

Michael:

Yeah. Amplifying our superpowers. When I think about automated excavation machines, we think of testing tools in that way somehow or other as something that can replace manual labor. And it is absolutely true. If you’ve got an excavator, that excavator can do a whole lot more work than a single person with a shovel. 

So now we can have one person operating one of these complicated bits of machinery doing the work of dozens and dozens of people, but testing isn’t like that. Testing isn’t like that. Unless what we’ve already done is we’ve turned testing into this sort of rote… unless we’ve already mechanized the human performance of testing, until we’ve taken an idea of how the testing is just this ordinary mechanical thing that we do, then we can make human testers go away. But my contention is that in that case, those were mechanical testers placed inside the body of a human, which is a bad idea to start with.

Instead, think, okay, supposedly we’ve got a really interesting tool that could help us with coding in the qualitative research sense, not programming, but classifying. Supposing we got a thing that could classify things to make a researcher more efficient, we wouldn’t be talking about getting rid of researchers. We would be talking about accelerating the process of classifying things, such that a researcher could analyze the data in more sophisticated ways. Now, this sounds like a really good idea and to a certain degree, it is. It’s kind of cool to have machinery look over large volumes of data and classify it in a very accelerated way. But of course, my tester mindset says, “Huh, what could possibly go wrong with this?”

And here’s how it could go wrong, if the data is in some way bad, like for example, if there are bits of the data that are being omitted, if there are bits of the data that are outliers in some sense that we want to be a little bit more skeptical about, if the data is in a form that is unsuitable for this kind of processing, but the machine tries to process it that way anyway, if it is time series data. One of the things that happens, I did some work with a company that was involved in some neuroscience related research. And so they would be taking signals from EGs, from the sensors, you hook up the people’s heads, and they would be trying to analyze it. Well, one of the first things that the neuroscientist/data analyst told me that was a big problem for this was that the time series are different.

Some people are taking measurements every fifth of a second, some people every 10th of a second, some people are taking measurements every eighth of a second, so that stuff has to get normalized. But when we normalize the data in order to get it to a 10th of a second resolution, we have to take those fifth of a second measurements and extrapolate. Did we have to take those one eighth of a second measurements and we have to normalize them using interpolation somehow or other. So the data gets fuzzy. Is that fuzziness important? So apart from all of the data wrangling that we have to do, we also have to do data analysis to ask ourselves the question, well, could this go wrong in some sense? 

I noticed in the last day or two, MIT has just withdrawn from access to AI researchers, this whole body of data, this whole huge data set, because it realized, surprise, surprise, that it’s systematically biased in various kinds of ways that would lead to misleading results.

The stories, the warnings that we’re getting now, they’re the famous ones about Amazon hiring profiles using machine learning and AI to select just the right people and surprise, surprise, the right people are just like all the GUYS, all the white guys who work at Amazon already, and not like different cultures, different ethnicities, different backgrounds, different genders, different gender preferences. 

The AI has noticed these patterns. Now, of course we can correct it. We won’t tell them specific things about anybody’s race, we’ll just give it the data and we’ll leave that bit out. So what does machine learning do? The machine learning finds patterns based on addresses where people live, and of course, people of certain races and ethnicities live in certain places and other ones live in other places. So it’s a really, really good way of fooling ourselves if we’re not super, super critical of our datasets.

Federico:

There you have an example of the realization frame!

Michael:

Right, right, right.

Federico:

… and you can see that it’s a social problem also.

Michael:

Exactly. So we’re aware of the famous ones. There’s stories about people who can’t get mortgages. And I know of at least a one tester who is reporting to me that she, a black woman, has a much, much harder time getting approvals for loans or mortgages, or a new car loan or something like that than her husband, who is a white guy. 

We got to start listening to people like Ash, because they have direct experience on one level of that, but of course, one thing that Ash is not particularly much these days is poor. So she’s well-off and I’m well-off as such things go. So not to question Ash’s empathy in any sense, but we do not have the daily experience of being a poor person and we don’t have… We’re not going to get rejected on the basis of the fact that we’re just poor.

So even the people who are super, super sensitized to this sort of stuff, because of their circumstance, because of their situation, I think they would be the first long before I would, but I’m catching up, I’m doing my best, they would be the first to point out, “There are things that I got missed on, there are things that other people could get missed on. We got to listen to those people.” 

That’s really super important. That one aspect of expertise that is, I think, critical: a real expert is humble about his or her expertise. And developing humility is sometimes a bit of a struggle for myself and it’s certainly a struggle of the people that we see leading tech companies. They’re very, very confident folks and they have a right to be confident, they’ve accomplished some amazing things, but a lot of their accomplishment is based on where they came from and their position and a lot of their accomplishment is based on luck.

Federico:

How do you mean?

Michael:

Oh, well, here’s an example. This example comes, for me, from Nassim Taleb and work associated with that. Imagine year one, I’m going to choose this number specifically because it rounds down so nicely in ways that I’m familiar with, we get a bunch of people who are stock analysts and we evaluate them based on how well they performed against the market. 

Well, on average just by sheer chance, half of them will outperform the market and half of them won’t. So year two, we’ve got 512 people. We started with 1024, the first year, and year two, we’ve got 512 and of those 512 that year, half of them outperform the market and half of them don’t. Half of them lose money. So in year three, we’ve got 256 of them and of those, half of them outperform the market, half of them don’t. Well, after 10 years, we’ve got one stock analyst who has overwhelmingly 10 years in a row has outperformed the market and that person is regarded as a genius, but they’re not a genius, they just got lucky.

For all of the people in Silicon Valley and other entrepreneurial positions around the world, there were lots and lots of entrepreneurs who ran into some kind of good luck and others who ran into some kind of bad luck, circumstances beyond their control. For example, somebody who launched an initial public offering in January of 2020, has one experience, and somebody who tried to do it April 1st, 2020, going to have another experience.

Federico:

Yeah, I guess.

Michael:

We’ve got to also look at the degree to which our experiences are based on good fortune.

Federico:

I think something you just mentioned, it’s important to be self-confident, but also to keep asking questions even to yourself, or this is also part of our curiosity as testers and our habit to continue learning. Right?

Michael:

Yeah. And it’s really funny, James and I were developing code recently, and as developers of that code, boy, were we confident that there were no problems in it. And it’s really, really hard for us as testers to look at something that we built and to say, “Okay, how could we go wrong? Well, here’s a few obvious things. Oh, let’s check for those. Oh, good. We’re fine,” like that. And of course, as soon as we started to use this product in a… Fortunately before using it in a class, I used it in a webinar, in a workshop and it’s failed spectacularly.

It was hilarious. It was really wonderful because it was such a great lesson on how your builder mindset requires you to, in a way, requires you to drop humility because you want to get something done. You want to achieve your ambition.

There’s a wonderful thing, I keep coming back to it. It’s from a terrific movie called General Magic, which was about an attempt to build smartphones in the 1990s, in the early 1990s. There was a vision for this thing and you can look at the drawings and it looks for all the world like an iPhone, 1992, 1993. And I was working close enough to people from that part of the world that I heard about it. James, at that time, he had been working at Apple and was working at Borland I think by that time. He was close enough in that community. He actually knew people who went there and maybe he was even invited there or something like that, but it was a project to build the first smartphone and it was extraordinarily ambitious. It was a really cool idea. 

They had amazingly smart people working on it. Bill Atkinson was working on it and had enormous amounts of investment. It was a mind blowing thing. And of course it blew up. The project never shipped. There were a whole bunch of issues associated with it. It involved a large number of companies. They had to get the telecoms in place in order to agree. And of course these are big stodgy companies, they were even more stodgy back then than they are now and they wouldn’t agree on stuff.

So in the movie, in the documentary, the guy who was the principle visionary for this project says something really wonderful illustrating the difference between the programmer builder mindset and that of more reflective and sober people. So there’s a guy that, Mark and Andy and Bill, Mark Porat is the guy who is eventually going to deliver the quote, someone is reporting that Mark and Andy and Bill together decided, well, this is the person to manage the engineering team. That’s what they wanted me to do. And then there was this one day where I came to talk to all of them at once and they say, this is the engineers, the engineering team. They say, “Oh, we don’t need a manager. We don’t want you because we don’t need a manager. Our leaders are Andy and Bill. That’s what makes this place great is that we don’t have managers.” And then one of the guys who wasn’t an engineer said, “Yeah, we can be engineers without a manager and we know what’s best. Managers are just going to get in the way. We don’t need program managers. We don’t need any of that stuff. We’re just going to make it happen.”

And then Mark Porat, who is the leader of the vision and who is just a delightful guy to look at in this documentary. He’s clearly an older, wiser man, very smart. Not in the tech industry anymore as far as I know. He says there was a fearlessness and a sense of correctness. 

No questioning of, could I be wrong? None, because that’s what you need to break out of Earth’s gravity, you need an enormous amount of momentum. And that momentum comes from suppressing introspection about the possibility of failure. And that’s why testing is such a difficult job because our whole world is about anticipating the possibility of failure and investigating where failure is happening. 

Looking at how failure could happen and gaining experience of the ways in which the product is failing to do what it’s supposed to do. That’s a socially difficult thing.

Federico:

Yeah. Totally. Helping others to continue asking the questions or having doubts about what they are trying to achieve.

Michael:

Yeah, yeah. That’s it. Our job as testers is to preserve doubt when everybody else around us is sure of something.

Federico:

Michael, I could continue talking for hours with you.

Michael:

Certainly, I could. Definitely obviously. But it’s a delight to talk with you though Federico. I had such a wonderful time in Montevideo and with your people and with yourself and drinking that stuff from that big coconut.

Federico:

Mate?

Michael:

Yeah. In fact, we enjoyed quite a bit of mate, the Uruguayan experience. And also did we ever eat well, for you it’s normal, but for me it was really special.

Federico:

It was a very intense week I remember.

Michael:

It was. Lots of ideas, lots of conversations, lots of discussion.

Federico:

I really enjoyed that. To try to wrap up, I would like to ask you if you have anything you would like to invite the listeners to do? I know that you are offering the Rapid Software Testing course online.

Michael:

Yeah. Now, I know that we’re recording this at one time and the distribution of it’s going to be later, but I think there will be one or maybe two classes that this will be in time for. 

One would be Rapid Software Testing Explored which I will be teaching oriented towards European and UK time zones. So I’m going to start at six o’clock in the morning my time, which is Eastern time in North America, that will be September 15th to September 18, four-day class, five and a half hours, three 90-minute webinars each day. 

And James will be doing Rapid Software Testing Applied, which is a whole ton of fun. Rather than being a survey of Rapid Software Testing, the concepts and the ideas behind it and little exercises focused on one thing to another, Rapid Software Testing Applied is a three-day deep dive into a particular product through the lens of Rapid Software Testing.

Federico:

Interesting!

Michael:

So those two things are on the schedule for now. That one is September 23rd to September 25th 2020, just in case people are looking at this in the archives. And then Rapid Software Testing Explored is the 15th to the 18th of September. But basically look at my site, (www.developesense.com), look at Jame’s site (www.satisfice.com) and our schedules are up there. And we will have classes being taught in a way that’s weirdly more accessible than they used to be because now anybody from anywhere around the world can join anytime.

Federico:

This is the advantage here.

Michael:

What you lose on the swings, you gain on the roundabout sometimes. It’s not the same as the in-person experience because there’s something really magical about being in the same room with people as we’re all finding out just now. People used to complain about meetings, well, boy, meetings in Slack and Zoom and all that are nothing like the real thing.

Federico:

Nothing. But still I will share the links in the notes of this episode.

Michael:

Perfect. Great. Thank you.

Federico:

I took this course in Uruguay when you were there and I really enjoyed it. It was amazing. All the things that you shared, the real experiences, critical thinking all the time. Those were like three very intense days in terms of thinking all the time, and this is something that I loved about the course and I really recommend it.

Michael:

Oh, well, that’s lovely to hear. Thank you. That was a great group too… I always admire people who are taking the class in a language that is not their native language because that’s just like this whole extra work that is going on. So it can be a little bit of a whirlwind.

Spacing it out over four days is possible now, it’s easier, of course, these days, again, because of that. It causes me to wonder maybe what we should do when everything gets back to something closer to normal, maybe we should just become tourists and do half days of classes over several days and enjoy the time that we’re spending in places. And that will give time for ethnography and detailed examination of how people are testing, do a class in the morning and then some research in the afternoon. It’s positing all kinds of possibilities.

Federico:

It’s a good combination of experiences.

Michael:

It would be. And we could do that with companies, at least, it’d be a little bit harder for public classes, but with in-house work that could be super, super valuable for everybody concerning you too.

Federico:

Let me know. Michael, thank you so much. Let’s do this another time again. I’m going to say it was great and stay safe!

Michael:

What a grandpa there.

Federico:

Haha Adios!

Michael:

Adios and gracias!

Did you enjoy this episode of Quality Sense? Explore similar episodes here!

Recommended for You

When to Automate a Test?
Software Testing Tools We’re Watching Out for in 2020