Blog

Quality Sense Podcast: Ian Goddard – Testing Virtual Reality Software

by December 2, 2020
Software Testing
0

In this Quality Sense episode, host, Federico Toledo chats with Ian Goddard, a very experienced tester and public speaker in the UK. He’s been working in software testing and automation for products related to broadcasting, video streaming, and most recently, virtual reality using your mobile phone or specific devices, such as special lenses.

Episode Highlights

  • How Ian got started in software testing
  • Automation and challenges of testing virtual reality software
  • Testing tooling’s need to catch up with VR technology
  • Skills needed to be hired as a VR tester

Relevant Links:

Listen Here:

Episode Transcript:

(Lightly edited for clarity.)

Federico:

Hello, Ian.

Ian: 

Hi.

Federico:

Thank you so much for accepting the invitation and I’m so happy to have you here in the show. How are you today?

Ian: 

Thank you. It’s wonderful to be here. Not too bad. Not too bad. Thank you, muddling through.

Federico:

Perfect. My first question for you, in order to understand a little bit more about your background in testing, can you tell us how did you end up working in software testing?

Ian: 

Yeah, I suppose anybody in our industry, I think we all can align with the idea of being an accidental tester. I think, if you ask many people, they’ll give you that answer and I’m going to be one of those people as well. Probably-

Federico:

That’s the most typical answer I get.

Ian: 

I don’t think I’ve ever met anybody that’s actually told me, “Yes, I wanted to be a tester ever since I was a small child.” But I tend to class myself as an accidental-accidental tester because when I was a student, as most students do, I needed money, and I went looking for a job that would fill some of that time and provide me with some money and I answered an advert on… We have a site in the UK called Gumtree, it’s very similar to Craigslist in the States. I answered an ad for a television tester, and I thought, “I can do that. I can test televisions. It’s on, off. How hard can it be?”

And then I turned up, and it was actually a job for one of the largest conformance testing houses in Europe and APAC and I just fell in love with it. It was, to start with, some very basic conformance run test scripts, and as it went on, it blossomed into the world of QA and what you can do around it. So, yeah. I’d say I stumbled into being an accidental tester, and from there, for better or worse, haven’t left since.

Federico:

So, I guess it was more challenging than what you expected at the beginning.

Ian: 

Oh, for sure, yeah. I was definitely, as a student, just wanted something a little bit easy, a little bit more money to go out over the weekend. But it turned into a whole big thing and really just the learning curve that came out of that was insane because I was expecting plug in television, switch on, that works, move on to the next one.

Federico:

Now, you’re working in a company where you’re testing some VR components, right?

Ian: 

Yeah. Yeah, absolutely. It’s been quite a journey to get to this point, but at the moment I work for a company called MelodyVR. What I do at the moment is I head up their QA team, I run the automation side of things as well. What we make at Melody is our product is an immersive live music app that allows users to be part of music performances. So artistic shows, music gigs, festivals, and it’s all part and parcel of the unique VR point of view. What that means in real money is that you can use our application to watch music performances within the VR space. So it’s across a number of platforms, but fundamentally it’s, you can use our platform to watch live music within the VR space.

Federico:

Have you met any artists working with this? I mean, now that you have to interview users in order to understand that.

Ian: 

Indeed, I would love to say that I have. Unfortunately I’ve joined the company during all the lockdown stuff that’s been going on, so unfortunately, I haven’t even met many of my colleagues, so it’s been difficult to meet artists but I’m hopeful. We’ve got, got some big artists lined up so I’m very hopeful that at one point, as a QA professional, I’ll have to be on site and I’ll have to meet them.

Federico:

In order to let people who have no experience with VR, like me, I haven’t tried any glasses or any devices… Can you explain how the product looks like or how to have a better idea of the product you’re testing?

Ian: 

Yeah, for sure. It’s a difficult one to pitch because it’s very similar to just the idea of having video content coming to you. So very similar to the idea of the Netflix and Amazon Prime and those video servings that you get. Instead of having them on the 2d screen, so you have it on your television and your mobile and things like that, what you have in the virtual reality world is this concept of the 3d, 360 degree, space around you. And the idea that you take the content that you want to watch, and you project it onto, it’s called the barrel around your space. Think about a bubble around you and when you immerse yourself into that world through headsets, or another way of taking the mobile application and you can put them into little tiny headsets that you can manufacture that cover the eyes. You shut out the reality in the world that you’re inhabiting now and you put yourself into this 3d world so that your eyes and your senses all believe that you’re part of what the content that you’re watching.

Ian: 

So what we do for Melody is we actually have cameras set up at various points on the stage or around the festival venue and… It’s difficult to describe, but imagine a big balloon of cameras. And it sits on top of a pedestal and it takes, takes images from all the way around, all 360 degrees and then we stitch all them together and you as a viewer inside the immersed world can view them in any space that you would normally. So if you go and see a gig and you really like standing at the back, you can stand at the back and you can look around and you can see the doors and you can see the bar. But if you really, really want to get close to the artist, we’ve got cameras on stage, that’ll put you right next to them or next to the guitarist or the DJ or the drummer or whoever you want. And you can still position yourself where you want on the stage or around the venue, and then interact in that 360 degree space by looking at whatever you want to look at. I hope that answers your question.

Federico:

Yeah. Yeah. Yeah. And it seems like something really cool to try. But my next question is how do you test that? Because when you were explaining, I started to think about multiple scenarios and boundary values and different situations that can go wrong in the user experience. So how’s your approach to test this type of application?

Ian: 

The approach to this is, it’s got to be fairly practical because you say there’s so many edge cases that you could just pick up from hearing that. Most of what we’ve got here is video delivery. So there’s a lot of the standard things that you’d come across with video delivery that are big players in what we test here. So things like latency, huge problem for any live video. This is something that we have to deal with, we put a lot of live shows on it’s still a big problem in the video world.

AV sync is a huge one for us. Primarily because, if you’ve got something like a sporting event or things like that, the AV sync can be slightly off and it does not matter so much because naturally you’re further away from the action you don’t necessarily pick up on it. When you’re talking about music performances and putting the users very, very close to the artist, you can pick up very quickly when AV is out of sync because you can see them talking or interacting and playing the instruments and that’s a huge problem. So that’s a really big one for us, is the AV sync. As well as that feeds into the general idea of the quality of experience that any video streaming has which can be a huge problem. What we call HD rates these days are mushy and annoying. They’re not necessarily the best quality that you can actually watch things in. But we have a lot of those standards, a standard way of doing it.

Specifically for the VR world, there are a lot of problems that can present themselves as well. And I think things that are very specific to that VR world, I mentioned we have a bunch of cameras on a big blob. There are blind spots within those camera stitching, it’s called whereas we put all those views together, there are blind spots for the cameras. So you have to make sure that the artist is well prepped that he doesn’t go drifting between the two cameras spots, because you can be the best stitcher in the world, but it’s still, you’re going to see a little divide and worst case scenario, you’re going to see a person split in half because they’re two different camera angles. So it can be really, really tricky there.

“I suppose it would be remiss of me to not mention the elephant in the room with VR, which is always motion sickness. People are very, very worried about that. And rightfully so.”

I think for MelodyVR, we’re quite lucky in that the way that we present the virtual reality is in something that’s called three degrees of freedom. What that means is that you have, within the virtual reality world, you can move around the rotational axis of your head. So you’ve got up and down left and right, and you’ve got… I’m always going to get this wrong, I think it’s the “yaw” which is the tilting your head from side to side in the 360 space. That is a lot better for the motion sickness stuff, because you’re not moving.

Motion sickness comes about when you have a disconnect from what your eyes see to what your body is telling you is happening. So when you introduce something called six degrees of freedom, that involves traveling movement along all three of those axes we just talked about. And in that case, that’s when you can think about, six degrees of freedom has the environment moving around you. When that happens, your body has a reaction that says, “Hey, I’m not moving, but you think I’m moving, there’s a problem.”

Federico:

Something weird is happening.

Ian: 

Yeah, absolutely. So that’s where most of the motion sickness idea actually comes in, it’s because there’s a disconnect between your eyes and your brain and your body. With Melody’s products, you’re very static so we don’t actually introduce too much movement. We have a little bit of movement on some cameras, but it’s not a regular thing on every camera so you don’t end up with this kind of disconnect. Sometimes in the VR world, you can have that problem, and this is one of the things with the six degrees of freedom applications, you often have peripherals like hand signals and things like that ground are you a little bit more in the environment.

The other thing of course is people come up with all these sorts of lovely space age, matrix style things, treadmills that run in every direction and harnesses that are antigravitic and all that stuff, which you see at all the trade shows and things, and looks fantastic, but I’ve yet to see them deployed in any anger anywhere. So for us, motion-sickness is not too much of a problem, but fundamentally it’s the video side of things that’s the main testing area that we need to look at.

Federico:

Within all the approaches for testing that you are considering for your testing strategy, are you also considering some automation? I don’t know about the different tools available for verifying, because when I’m thinking it’s probably the hardest part is to add validations to this type of system, right?

Ian: 

Yeah, absolutely. It certainly is something that we want. You speak to any QA professional and they’ll probably give you the same thing, “Yes. We want automation to be in there.” The thing is with automation for the VR world, and I’d say this is again mirrored in the video world as well, is there’s a lot of things that are very, very hard to automate there and next to impossible a lot of them. We mentioned quite a few things being quality of experience, and that can be very subjective, which makes it, as you say, it’s very hard to verify what’s actually going on. You can have set parameters that you might want to check.

So I mentioned the HD rates and you can verify that you’re getting what is defined as an HD rate being reproduced in your application, that can be programmatically done, and that sort of problem, with automation can cover that. However, just because it programmatically says you’re getting the right HD rate down or whichever profile level that you’re actually seeing the video at, that can actually translate into not a very good viewing experience for a number of reasons. So yes, programmatically, you can come in and do automation on that level that tells you exactly what’s being reproduced in this virtual reality world in the video scope. But actually, is that going to give you what you want at the end of this? Because yes, you can automate a lot of, unit level stuff, but is that really going to tell you how the end user is actually viewing your product?

So end-to-end testing is incredibly hard to get the automation stuff in there. It doesn’t help as well that the level of maturity for the tooling that’s available is quite low. So you’re almost forced into the idea of creating proprietary tooling just to fit the niche that you need. I’m a big believer in putting in as much automation as you absolutely can. And we may say, “Oh, you only get a percentage of things automated.” Yes, but that’s a percentage of things you don’t have to manually test. So the advantages I think outweigh not putting them in. The VR world can be tricky.

Federico:

Yeah. I am thinking now that this is something that typically happens with all the tooling around testing. First, you have the technology and maybe a couple of years after, you have tools that help you test this specific technology. So probably the same is happening with VR.

Ian: 

Yeah, I completely agree with you. When I first started my journey in the QA world and automation was still a bit of a dark art that only certain specialists really knew. And you had to have a developer hat on in order to run automation as a QA person and it was very secretive and now certainly web automation, and I’d probably put mobile into that too, is becoming so much more accessible that many, many people can pick that up.

Within the VR world, I think we’re at that place now, like you say, it’s still new, it’ll take some smart brained people to come in and build some tooling, to demonstrate that as possible, I think once you’ve demonstrated as possible, then people can make it better.

And I certainly, for all of my automation stuff that I do is I’m a big believer in open source. And it’s difficult for companies to build proprietary tooling to make that jump over to let’s make this open source so everybody can do it because it gives you the advantage in the small world that we’re in, where nobody has it yet. It’s difficult to convince people to move into that space. So the stuff that we build for VR, I would love to be open-sourcing, I’m a big believer in paying it back, but who knows? We’re not quite at that stage where we’re talking about rolling out the next Selenium just yet, but hopefully one day.

Federico:

Maybe in the future. You mentioned at the beginning, you have to consider all the things for the broadcast. We’ve been working on plugins for JMeter in order to automate a performance test for a broadcast but there are new protocols appearing for broadcasting video. Like what I mentioned before, you have the protocols, you have people working with them and now we are working in the tools to provide you the ability of doing a performance test.

Ian: 

Absolutely. We’ll get there. And it’ll open up the world. The video world is quite closed up at the moment, but so as I say, Selenium was quite closed off a few years ago and now it’s opening and we’ve even got competitors for it and that’s, that’s amazing to see.

Federico:

Yeah, totally. I have another question, which is because you are in a very niche industry, so I guess it could be difficult to find people with the skills needed to test your product. So when you think about scaling up your team, how do you onboard someone? How do you train them? Which skills are important?

Ian: 

Yeah, absolutely. It’s a very interesting question and actually one that I am dealing with and HR dealing with that as we speak. We’re a growing company and we’re only getting bigger from here so we’ve got to scale up. It can be difficult, the idea of finding the right person for the job can be… I don’t like to pigeonhole people and say, “Okay, well, this person has got automation experience. So they’re going to be really good at doing the programmatic side. Or this person’s worked with the VR world so they’ll be really good at…” People can be different across whatever industries that they’re working in, but for us at the moment because it’s such a heavy leaning on manual testing, that’s where we’re mostly searching for our candidates, is people who have experience. 

It’s wonderful to find people who have experience in the virtual reality world, but there are not very many of them and as I’m sure you can be aware. So finding those people is great, but it can’t be a prerequisite for our hiring.

We’ve had people interviewing from a wide range of… A lot of people from the games testing world. They do have a lot of transferable skills there, they already know the VR space a little well, if they don’t necessarily know it as being the virtual reality space, the kind of games inhabit a virtual reality space within a 2d screen or as it were. So they are already there and they’ve got experience with some of the engines that we use to generate these spaces. 

Finding the right people to interview can be tricky, but that’s where we’ve had a lot of good candidates actually, to be fair and it’s been really interesting. From lots of different backgrounds as to how we pick people up, from broadcast gaming, straight web development, we had a couple of developers actually, who were moving across, making that jump into the QA world, which is wonderful to see because it’s usually the other way around.

But when we start to scale up and we’ve got people in, the training has to be quite a soft approach, I think. I don’t believe anybody’s going to be the best and know everything that’s going to happen with a new company within two weeks, especially given the situation that we’re all in for the last year, it’s difficult to onboard people into that world. So I don’t demand people are suddenly working on tickets after two weeks. I think that’s completely unreasonable.

Getting a VR device with an application in somebody’s hands, as early as possible, is one of the best things that we can do. Not just because it could be somebody who has never used VR before, or it could be someone who’s got extensive use of VR, but has no idea how our application works. So giving them the time to, essentially, have a free play session. You say, “Hey, we’ve installed the app for you. Go and play with it, go and watch your favorite artists on there, have a play around, do the things that good QA do and poke everything until you break it and then you understand how it works a little more.”

Federico:

You understand the user in that because you become the user.

Ian: 

Absolutely. Yeah. And it’s something I’ve always said to anybody who I’ve onboarded to any team I’ve ever been part of is if you break it, that means that a user might break it. So don’t worry about breaking it. Don’t be scared to go and hit things and then, “Oh, I’m really sorry, I hit the button and now this is broken. And it’s that…” No, that’s exactly what I’m paying you for. I want what I want. So that’s how I approach things, I want people to just go and be the good QA that they are and break stuff, play with it, learn how it works.

The other thing, and I suppose I’ll make enemies on one side and friends on another side is documentation. Me personally, I like things being documented, but that comes with the caveat that they’ve got to be maintained because I have a friend I used to work with a long time ago, who when I asked them to update a confluence page, told me, “But confluence is where documents go to die.” And I think they might be right for a lot of them. But if you’ve got these things documented into a welcome pack and just how do you install the application?

Simple things as well like who are we? In this day and age where I haven’t even met half of my colleagues, who is in our QA team who is this person, this disembodied voice that you hear on the stand up every now and then. Just having a set of documents you can hand over to someone and say, “Here’s a bunch of documents. You should read them, not all at once, but if you want to come and talk to me…” I tend to try and make a lot of time for any new starters to say, “If you bother me during the day, it doesn’t matter. My entire focus is you for the next week or so. So I’ll get other bits of work done, but you are my focus.” So I guess a soft approach pairing with people and getting them up and running with the application is my goal for that.

Federico:

I guess that collaboration with developers is also key here and also establishing the machinations with the rest of the people will be a challenge nowadays.

Ian: 

Yeah, absolutely. It can be challenging in the best of times when we have before all of this, where we could actually go and talk to somebody at their desk. I know I’ve always been a big believer in, I hate the us versus them mentality QA versus dev, I hate that as an idea, I hated it when I operated on over the fence QA team and I hated it when I was trying to integrate into development teams. And now it’s even harder because you need to A), you need to know who they are, at least in the olden days when we could go and walk over to somebody’s particular area and know, “Oh, these are the front end guys. These are the guys.” You take note of their faces and you can see them now. Now it’s kind of, “Okay, well, who’s working on the front end?” Okay, well you need to talk to Alex, you need to talk to Steve. You need to list them out and actually direct somebody towards these people. So it can be very difficult to get them integrated in a way.

“I’m a big believer in QA. I hate to use the cliche, but the shifting left mentality and getting QA involved early is vital for the applications that we’re all developing these days.”

IAN GODDARD

If you don’t have us involved, we’re going to go back to the old school of where you kick it to us, we play with it for a while, we kick it back to you and then we say there’s problems and you kick it back and we bounce that around for a while and then four months down the line later than our deadline, nobody’s happy. So if we’re involved early, everybody wins in that situation. That’s what I want to do with anybody who’s on board is really getting involved early and say, “Hey, look, these are the people you’re going to work with regularly, get to know them. I don’t mind if you go and have meetings with them outside of the QA meetings or the ticket meetings or the workload or whatever. Just chat with them, become their friend, that’s one of the benefits of working with people. You get to meet new friends.

Federico:

Yeah. Yeah. A couple of final questions for you. One of them is if you have any habit that enhances your productivity that you want to share.

Ian: 

Yeah. I suppose there’s a couple that I would say have served me well. And in my time they are cliche, I suppose. But the big one for me, I’m a terrible person for taking on too much work. And I’ll say, my advice is learn when and how to say no. Because as a QA person, you can often get to take on more and more tickets because you think it will be quick and you think it’ll just be a nice little check and then it turns into a really big problem. And suddenly there are three other tickets that you’ve also said yes to haven’t had any attention today and now it becomes a spiral. Being able to say no, and diplomatically say no, can really help. It certainly has helped me. The other big ones-

Federico:

Sorry, I think it’s true. It’s a cliche, but it’s really hard to do it properly because I struggle all the time with this. It’s really difficult to say no, mainly when you feel passionate about so many things, it’s yes, we have to prioritize.

Ian: 

Absolutely. And I mean that the priorities of things can come down from any source to your product owner or whoever your actual boss or your QA manager or whoever. But at the end of the day, you’re the only real person who knows your workload. If any people who report to me say no to something, I know they’ve got a good reason for it. I like to give people the benefit of the doubt in that, there’s a reason you’re saying no and if there’s a reason I’m happy. I’d rather say, let somebody else deal with that task. Not a problem.

The other big one for me, and I think this is something that took me a long time to learn is, just accept the fact that it’s okay to ask for help. And that’s not just for a point of view. If you’re struggling with something, I found that this is a really good piece of advice for any QA person that’s trying to make the jump into a more technical role, maybe somebody who doesn’t necessarily have coding experience. Go and ask the developer who worked on that ticket just to explain to you how it works. Or honestly, if you just want to QA it quickly go and get them to show you the feature working. And it’s amazing how much you learn about the products and the coding and how things fit together, but also how quickly developers will QA themselves if you ask them to demonstrate it. The amount of times that I’ve said that to a developer and they’ve gone, “Oh, no, that’s a bug. Oh, that’s also a bug.” And then you sort of think, “Well, I’ve done my job and also I’ve learned something.”

Federico:

Yeah. I really liked this because it’s like we started talking about very challenging technical things but the key skills that you’re highlighting are very soft skills or professional skills as they call it. For me, the second one, it’s hard because you have to be humble in order to master this skill. 

You have to recognize that maybe you cannot do it alone, you need someone else’s help in order to achieve or to learn more or to… But the first step is recognizing I don’t know that and it’s okay.

Ian: 

It’s okay, absolutely. And that’s the thing as well, it’s difficult from day to day. You can have a really good day and be all over things and then the next day, those things just don’t make sense. As you say, if you can recognize that in yourself, which is harder said than done on a lot of occasions. But if you can, it really does help. I think.

Federico:

Totally. I can see in your background that you have a lot of books. So I would like, if you can suggest or recommend any book you like.

Ian: 

Yeah, certainly. I suppose, like all these other questions, I probably have several that I could, I could go to. From an industry point of view, I’ve never been one for reading a lot of books about the industry.

I find that hands-on learning can be more valuable than most books, but I would say that things that have impressed me from an industry side, and I know that she was featured on series one of your podcast, Lisa Crispin and Janet Gregory’s books, Agile Testing, and then subsequently More Agile Testing were both excellent reads. I was a big fan of them. So they’re probably my industry picks, I would say.

From a non-industry pick, I’m a massive sci-fi nerd. And I’m a big, again, making enemies, I suppose, but I’m a big SpaceX fan and the idea of going to Mars that makes me very excited, lots of testing required there, I’m sure. And I think a book that I would read or take for anyone is a book. It’s actually a series, but the first book is called Red Mars. And it’s a book about, it makes a couple of assumptions that we can get to Mars and we’ll get to colonize Mars and then it’s a future history. How that unravels, how do we go about the politics of Mars and their interactions with us? And it’s a big book, I think there’s three of them and I think they’re all about 600 pages. So settled in, but I’d say Red Mars is definitely one I would say to anybody, probably my favorite book right now.

Federico:

Cool. And we’ll share in the notes, the links to the different books you mentioned. To wrap up the interview, would you like to invite the listeners to reach out, to try your product or whatever? This is the time for that!

Ian: 

Yeah, absolutely. MelodyVR, I’ve never had to go to the website myself, but I believe it’s melodyvr.com. You can see what we’re doing there, some of the artists that we’ve got on the applications and some of the upcoming, that’s available on mobile devices, Android, iOS, as well as if you’ve got an Oculus headset, then please do go and go and get it. 

You can sign up for free. There’s lots of free shows on there. You can enjoy the product. There’s a lot of artists, I don’t know who they are, which I think ages me more than it does the artists that we’re getting on, but lots of good stuff on there and we’re doing lots of ticketed shows as well. So a lot of big name artists that come in for shows, you can come and see them.

And if you want to follow me, I think probably Twitter is the best place. I do tweet from time to time. Some of it’s industry-based, some of it is me ranting and some of it is me posting funny cat gifs. So that’s @IanGoddard88 if you want to follow me. Don’t expect too many exciting things because I’m not that exciting, but I do interact with everybody who talks to me so if you want to reach out to me, that’s fine.

I suppose, other than that, once we settled back into normal, hopefully I’ll be back at conferences and speaking. If you see my name and you’re interested in talking to me about anything, I love talking to people at conferences, please do stop me in the corridors and come see me if I am talking, but also stop me in the corridors if you want to have a chat.

Federico:

Amazing. Thank you so much Ian for your time.

Ian: 

Thank you.

Federico:

I will try the application, I’m very curious about it. And also I will think with a mind of a tester, how-

Ian: 

Fascinating. Let me know which ones are you going to break?

Federico:

Okay. Bye-Bye. Thank you.

Ian: 

Thank you. Bye.

Did you enjoy this episode of Quality Sense? Explore similar episodes here!

Recommended for You

When to Automate a Test?
Quality Sense Podcast: Mike Lyles – Testing & Leadership

Tags In
Open Source test automation
208 / 437

Related Posts