Quality Sense Podcast: Matias Fornara- Test Automation

Welcome to another episode of the Quality Sense podcast! For today’s episode, I had a very interesting conversation with Matias Fornara, he’s a Senior Leader at Abstracta and we’ve been working together for more than 10 years. We discussed everything Test Automation, how to ensure its quality, and much more. So get comfortable and enjoy the episode!

Episode Highlights

• Matias’ start in the industry and his transition to Software Testing.
• His take on how to continuously improve the quality of your test automation.
• Opportunities for improvement within a team.
• Matias’ Test Automation Manifesto.

Relevant Links:

Follow Matias on LinkedIn
Matias’s book recommendation

Listen Here:

Episode Transcript:

Federico:

Hello, hello Mati Fornara. I’m so excited, so happy to have you here in this space, in this podcast. I’m especially happy to have the opportunity to share this space with someone that I’ve been working now for many years. How many years we have been working together, Mati?

Matias:

Indeed. Thanks for having me, it’s really, really a pleasure for me to be here and of course an honor to be hosted by you.

Federico:

How many years already?

Matias:

Together with you in Abstracta, almost five but in the industry almost 10, more actually yeah. Let’s call it 10 so I feel younger.

Federico:

Okay. Perfect. So Mati to let the audience learn a little bit more about you please introduce yourself. Tell me your story and also how you ended up working in software testing.

Matias:

Sure thing. Well, I’m Mathias since you might already heard. I’m from Uruguay, I work with Fede in Abstracta. I’m a senior leader, much more focused on automation and performance but nowadays mainly focused on automation because I’m leading the automation efforts in Abstracta. And that’s it basically from me.

Federico:

Nice. And how you started working in software testing?

Matias:

Well, that’s a long story but I started in 2013, I started not as a full-time tester. I started in a big company where I developed, I tested, I was an IT support and then finally-

Federico:

So many hats.

Matias:

Yeah, so many hats. And finally I ended up working for a healthcare insurance, testing for the healthcare insurance company. And after that I moved on to another shop where I dedicated to myself to develop, right? Mainly airline and Java. And then I am in 2017, I started working in Abstracta fully, fully dedicated to quality.

Federico:

That’s very interesting, especially when you are one of those rare cases, when someone starts doing development and moves and transitions to software quality and testing. This is not the common scenario, right? In most cases I think I’ve seen people who started as a QA tester automator or something related to software testing and then moves to development or similar role, right?

Matias:

Yeah, totally yeah. Actually, automation is so much used as a mean to develop at the very end. But in my case, it’s like somehow I was always related to quality because great part of my developing experience let’s say, was related to a huge automation framework. And at some point, I realized that I should stop lying myself about developing and I should focus on whatever I have been doing by that time, that was actually testing. Right? Well, Abstracta came in at the right time, let’s say.

Federico:

And you like it so far, you’re happy with your decision on transitioning to software quality?

Matias:

Yeah, yeah. I would say that based on this long-term relationship that we have with Abstracta, yeah. I do, I really do.

Federico:

And Mati tell me, do you recognize an inflection point in your career, something that changed everything, something that you did, or something that happened that changed your perspective about software testing or changed your possibilities? I don’t know.

Matias:

That’s an interesting question. Yeah, I would say that many things happen that have somehow broadened my perspective. For instance, I don’t know, teaching software testing, performance testing, automation, right? Also preparing talks, workshops or basically any kind of presentation, right? Those things have shown me that we need to be very careful on how we convey concepts requirements or ideas, right? Depending on the target audience and being aware of this, let’s call it challenge, right? Changed the way I approach software projects from a testing perspective. Regarding the inflection point, I would say that it will be the moment… And this is not a advertising, right? But it would be the moment when I joined Abstracta because I have had several leaders, of course, you are one of them. And friends who trusted me and believe in me in ways that I haven’t been able to do so far, right? That helped me to achieve certain goals, the trust that they gave me became into confidence and to start teaching, leading, and why not being here in this podcast today. Right?

Federico:

Nice, this is a very nice story and I think that when you try to teach or explain something to some other people is when you learn the most, right, you are one of the person who gets most of the benefit of trying to explain something to others and how important it is to have someone who trusts in you, right? So I know that this is something that you typically do in your day-to-day basis, helping others that you lead and trusting them, and helping them to grow. So that’s an important part of our careers.

Matias:

Totally, totally.

Federico:

So Mati today we wanted to talk about test automation. Also because we are preparing this talk that we are going to give together in TS QA conference. And we wanted to share some of the perspectives on things that we were discussing when preparing the talk. And basically the main idea behind the talk is that in most of the cases we’ve seen that people start with test automation just trying a new tool or start doing some automation. Okay, but after a time you have a lot of test cases automated maybe and even running in your CI engine, whatever. My question is, how do you keep the quality of the test harness of the test automation, how to continuously improve the quality of your test automation, how to review the test strategy and check that it’s giving you what you were expecting from the automation?

Matias:

Yeah, that’s a very interesting question that I would say that it has multiple answers but to be concise, I think that there are like two different challenges here that we should address. One, I would say is related to goal quality, right? And which practices we use to review every increment that our framework has. There needs to be an agreement let’s say, on how we as a team develop our automation framework, right? In my opinion, that needs to be clearly documented as well as the responsibilities within the team. We need to generate that sense of accountability in the team to keep certain quality standards let’s say. And regarding the second challenge is much related to the automation strategy, right? And how that strategy fits into the whole testing strategy. Right? We need to be constantly looking for the team’s feedback. We need to understand if we’re adding the right value to the right people. What do manual testers need from us? What does the projecting owner need from us? I think as in any development process, we need to seek efficiency, right? So we need to know what are our manual tester is doing and how are we going to help them. Are we going to automate everything at the API level? If we do that then, what we should cover at the UI level to avoid duplications, that’s the kind of thought that we should have, right? I know that we talk a lot and that’s something that you read in every single article but we talk a lot about testing pyramid, automation pyramid but actually when it comes to real life, we tend to forget very quickly about these concepts. And we end up trying to automate everything and causing a mess very quickly.

Federico:

I was thinking exactly about that while you were describing the challenge because we typically speak a lot about the best practices or the test automation pyramid model and things like this. But in the practice, how is your experience when you approach a new project and you see if they are following that, maybe even when they say that they are trying to follow that strategy, do you think it’s something that people keep in mind after a while?

Matias:

I think that it’s a wish, right? And it’s something that we think at the very beginning of the project. But if we don’t do anything to, well as I was saying, right? Document the decisions that we make regarding best practices regarding responsibilities, right? When the hard times comes, let’s say we have multiple features coming in our backlog, we have to automate multiple tests whatever, right? It’s very easy to forget about that and if we don’t have like a source of truth, right. To go to this in these moments of difficulties, right? Then we tend to get lost, right? And we try to cope the expectations and the necessity we have in that moment in whatever way we can, right?

Federico:

It’s a reference model, right? So the idea is to take it into consideration when we start but also during the project to check frequently if we are more or less doing what we said we were going to do, right? Sorry, you were going to say something. Okay. So another question is when you join a new project, what are the things that you will try to find out first in order to have a better understanding of the status of the project or the context to align the automation strategy?

Matias:

Well, I have tried to find like a pattern for these occasions, right? And the main thing that we’ll try to do is to understand the delivery model, right? For the project so once I understand how does it work, it might be a super complex delivery model or not, right? But what I try to do is well, to understand where are the pain points, right? That this delivery mode can have because most of the time the answer is not, “Okay, let’s automate,” and that’s it, right? We need to know where doesn’t make sense to automate. It’s not like, “Okay, please come automate my process and make them faster.” Right? This is this belief that solves our problems, right?

Federico:

Just to our efficiency, add some efficiency to my testing, please.

Matias:

Exactly. Let’s add this robot that makes things better, right? There’re always, or at least in my experience, there are always methodology improvements that we can do. Communication issues that we can address and maybe one of the most common scenarios, which I face pretty often I would say. Is a complete lack of manual testing, right? So in those cases we need to create, let’s call it a testing base, right? Something to work with in terms of testing because at the end of the day what we are looking for is to improve the quality, right? And automation is just a means to an end and not an end itself for me at least, right? And it’s super important to understand why do we want automate, right? And what do we want automate to achieve that goal?

Federico:

So it’s not a myth, it’s a reality that there are some companies or people in some projects where they are trying to automate everything and they are trying to get rid of manual testing?

Matias:

Totally, totally. And maybe you see these motos, this company, motos like, “Oh, we care about testers, we care about quality,” but when it comes to the real-life let’s say, they try to develop and they try to rid off this problem that quality is for them, right? Just to, “Okay, let’s throw this development into a box and let’s wait for the output to be something with quality,” right? And they don’t even understand that by just automating you’re just validating what you already know, right? And not discovering anything new about the product.

Federico:

Yeah. So again the strategy should include both things, right? automation and manual efforts, perfect.

Matias:

Yeah totally. It’s super important that we understand that we are not alone in this ship, right? It’s a test strategy that has within itself, automation strategy to achieve something. And sometimes it’s like we don’t know what do we want to achieve, right? In terms of quality, what quality means depending on our context, right?

Federico:

Yeah. So what about the team and the way they interact with each other, is that something where you typically find opportunities to improve?

Matias:

Yeah absolutely, absolutely. We need to review how we collaborate, right. Asking questions to keep it relevant for the whole team let’s say, right, and there are a couple of common questions that you it’s good to have on your pocket let’s say. For instance, how are we defining what we automate, right? Who is involved in that decision? for instance, that’s something that I tend to ask to the teams very commonly. Why, because it’s important to have fluid communication between, I don’t know, Product Owners, testers, developers and automators of course, to define it, right? It is essential to know that we are taking the best approach we can when it comes to well, to defining what to automate, right? And then we need to understand how other aspects of collaboration works for instance tools, right? Are these tools that we picked let’s say, I know task managers, test case managers. Well, I know whatever, right? Are these tools helping us to give visibility to the automation, right? Or we should be wondering, are these tools enough for us or these tools are hindering our way to what we really want in terms of quality, right? We do understand if we’re paying attention to these tools as a team, right? Because at the end of the day we’re not working alone in the project as automators, right? And of course in terms of communication and collaboration, right? There is documentation and a great part of the task of an automator is to well, to verify that our test framework is well documented, right? Because this frameworks documentation communicates and makes it easier for our project to scale, right? So it’s important to document certain aspects, right?

Such as, I don’t know the setup or how we execute the test or which environments are we using, how are we generating the reports or basically any technical aspect that we have already resolved for instance, right? And that we know that it can turn into a possible setback let’s say, for any new automation in the future, that’s something that we should document. We should even think if you want that the framework should be useful for anyone by anyone actually, right? By, I know a developer or any other person that might want to use the framework in our absence, right? That’s what we should be looking for with our documentation. And finally one thing that may sound simple but it sometimes cost us a little bit of time in terms of when it comes to automate, right? It’s the reports. We should be looking into the reports, and mainly we need to understand if those reports are providing value to anyone, right? Are anyone in the project that we are making some kind of decision based on that report, right? Are they really necessary for instance, because we have encountered times where we are creating super complex reports and the only thing that we really need is just I know, green lag in our pipeline saying, “Okay, all the tests have passed.” And that’s it and that’s all the information that we need to decide whether we should deploy to a certain environment or not, right?

Federico:

Yeah, it’s amazing connecting with something that you mentioned at the beginning. It’s amazing how most of the challenges that you are mentioning now are very similar to the challenges of developing applications. I mean, we shouldn’t forget that coding tests, automating tests is also a programming activity. And all the best practices and recommendations and challenges that we have for coding, we also have them here and we should try to apply maybe the same solutions that engineers and developers apply in coding and application, right?

Matias:

Totally, totally yeah. I completely agree with that, we shouldn’t be doing anything different from whatever the developers are doing with the code base that is meant to production, right? I don’t know, there are lot of considerations, right? When it comes to, I don’t know, best practices, right? That we tend to think that those don’t apply to us but they do, and well as you said, right? It’s code, code is code after all, and we should be using the same version control practices that they are using, right? We should have a defined branching strategy, we should define how are we going to go review our pool requests, right? What are the considerations that we should have and again we mentioned documentation in terms of the framework. We should document of course, what kind of best practices apply to our development process because again, code is code but there are few exceptions, right? Few differences between well code meant to automation and meant to any other thing. Right, but at the end it’s important to establish more or less the same things like naming conventions, patterns that we should do, the same patterns that we should do, right? Page options, screenplay whatever, right? and then our common practice as peer review, right? That’s something super useful, super useful that’s not so commonly used let’s say. But it’s super useful because not only the person that receives the feedback let’s say, is learning a lot from that exercise. But also the person who makes the review, right? Learns of how a problem is resolved by a different person right? And it helps you to broaden your perspective in a way.

Federico:

Yeah, absolutely. So Mati you recently came up with something that I know you have been working with all the test automators Abstrata that you called the test automation manifesto, right? And I think you are trying to cover all the aspects that you were mentioning right now. Is like a working agreement or something like this, right? Can you tell me a little bit more about that idea?

Matias:

Sure. First, I don’t want to attribute that idea only to me because that’s the product of a collaboration it’s completely a team effort. 

Based on experience, right? We have had what you already know, multiple projects, automation projects. And we have learned a couple of things and yeah we understood that we need to write down these things that were in our mind, right? That we take into consideration when it comes to automating, right? But well it was a common agreement that we used to have but it wasn’t written so we decided to do that. We create this so-called manifesto yeah, it has recommendations and well conventions let’s say, for everything, right? Since I don’t know, how to name a class, a method, a package, right? How is I don’t know a common branching strategy that you should use, a naming convention for the branches, for the future branches. For I don’t know how you’re going to call the variables that store the elements, like I don’t know, buttons, text, fills, whatever, right? Just to make it easier for us when it comes to review that code, right? Well the action of reviewing the code, right? Because we have this common understanding for us is very simple to say, “Okay, this is out of place, we need to correct it, right?” And yeah, yeah of course and that together with another practice is more related to I don’t know CICD for instance, where these tests should be run. If we are using a linter, how we should configure it. If we are using, I don’t know other tools for static analysis, like I like sonar queue, where this analysis should be run in our pipeline, right? It has like a very huge or broad bullet points of things let’s say. And yeah, it has been super useful for the new project that we have started after we create the manifesto.

Federico:

Yeah. I find it useful also for the customers, if they want to understand the automation maybe having that documentation, they can understand a little bit more why are we doing our test cases in the way we are doing it, or why are we orchestrating or I don’t know, the different practices are easier to understand for them or for new team members when the team is rolling. And yeah, that’s I think a great practice that you are all doing. But we are talking here about test code because we are considering all those test observation tools based on code, right? Like Selenium but there is a recent trend to use low code or no code or script list, it has so many names for test observation, right? So do you think that all these considerations and those practices can be applied also when working with this kind of tool?

Matias:

Yes, yes I would say so. I mean again, not in the same way let’s say, right? But there are concepts as modularizing, right? That still applies to this new sort of tools, right? This concept of trying not to reinvent the wheel every single time, it also applies here, right? Then it depends on the tool how that is resolved, right? But typically in those kinds of tools, you have ways of, I don’t know, creating modules from some kind of test, for instance, I don’t know if you want to modularize the login, you can create like a separate module that then you can reuse, right? And then this part of okay, naming conventions. There is a super important part, I mean there are tools that resolve this part of picking the right web elements. But some tools don’t do that and we need to have a concrete selector strategy, right? Or locators strategy and that is useful for this kind of tools and also for the code tools, right? It’s the same concept but again, I would say that the main concept that applies to these tools to these local tools is trying not to reinvent the wheel every single time, trying to reuse code, avoid duplications, right? And that’s super important when it comes to low code automate.

Federico:

Yeah and I’m thinking of other practices that also apply even, for example, code review, if you don’t have code and maybe you don’t have a pool request in your GitHub repository but you can implement some sort of agreement where before integrating these new tests into the pipeline, someone else should have a look, right?

Matias:

Totally, totally. I mean and that should entail since verifying that the test actually works, right? And it passes but also understanding if the test is doing what is meant to do, right? What it was assigned to do if it is validating the right stuff and the right moment if the assertions are correct, right? Those are the kind of stuff that lead us to have flaky tests, we don’t validate those kinds of things.

Federico:

Mati this conversation I think it’s full of insights, thank you for sharing all of this experience with us. I wanted to ask you a few more questions to wrap up this episode. One of them is if you had to recommend that one book, which one would you like to mention?

Matias:

Well, I have read a couple of books lately but I would say that one that maybe is worth sharing is well, actually a book that you get rid of when you move to the states and it’s called Sddhartha right? It’s a book from Hermann Hesse. It’s a very short book but it leaves you with this relief and happiness sensation after you finish it, right? Like it helps you to build life in perspective, to value the important things, right? In life and the importance of how the time passes, right? And well that helped me a lot in life in general, right? But also when it came to leading teams, right? To understand what’s really important, right. What is not only a job, it’s a person, right? And what it’s important for me, what’s the right message for me to convey. That’s not only getting the job done but it’s also enjoying the journey let’s say, and while doing this task that you have to do.

Federico:

Amazing Mati, thank you so much for that. And is there something you would like to invite our listeners to do or to reach out in social media or something?

Matias:

Yeah. I would completely recommend you to stay tuned to the Abstracta blog post. We are posting a lot of new blog posts, we have a new one that we release a couple of days before this podcast was recorded and that’s regarding Selenium 4. And we constantly posts since reviews from some kind of tools also how to improve your testing strategy and also sometimes, right? So yeah, please keep tune because we write tons of interesting things that I would say you would like to read.

Federico:

Excellent Mati, thank you so much. It was a pleasure to talk and practice our talk for the conference that we are going to be talking about these topics. Well, probably this podcast is going to be released after the talk. So please try to find the talk online.

Matias:

Great.

Federico:

See you soon Mati.

Matias:

Thanks for having me Fede.

Federico:

See you soon Mati. Bye bye.