Reviewing the newest code-less test automation tool on the block

If you’ve read our blog before, you may have noticed that we love to try out new testing tools here at Abstracta and share our opinions of them!

Recently, one of the tools we’ve been learning more about is a newcomer to the test automation space, Reflect.

According to its website, it’s a cost-effective, zero-code, zero-installation tool for recording and executing UX tests that are also easier to run and maintain.

We wanted to see if the tool lived up to the hype, so we scheduled a demo with the Reflect team to see it in action, playing around with JsPaint, and meanwhile, we were able to ask them any questions we had along the way.

In this post, we’ll share our overall comments about the tool and some of the strengths we see in it as well as what we hope they include in their product roadmap.

Reflect’s Strengths

Hover Tracking, Drag and Drop, etc.

The Reflect team told us that existing record-and-playback tools either don’t support hovers at all, or you’re given an option to enable mouse over/hover tracking at which point you get a ton of unnecessary steps in your test because it’s recording every hover event. 

So, they made it so that Reflect supports hover tracking that records only the hover events that are meaningful in the test. It has a click-and-drag/drag-and-drop/mobile swipe support that exactly replicates what the user does. Similarly, you can exactly replicate clicks/hovers/taps/swipes/drags which would otherwise fail in other tools.

The Browser Has a Pristine State at the Start of Every Test

Rather than in a Chrome extension, recording is done inside Reflect’s web app. Their team explained to us that the extension approach opens up a lot of susceptibility to false positives:

“If you’re logged in or otherwise have cookies set, if you’re behind a proxy, if you started recording mid-way through the test – these and other factors can all lead to a bad recording that fails on its first run. With Reflect, you record via our cloud-based browser. This differentiator itself lets us remove a lot of the factors that lead to bad recordings. For instance, we can guarantee that the browser has a pristine state at the start of every test.”  – Reflect team

This also allows you to use different profiles when testing. For example, to record a mobile test, you select the mobile profile and it’ll open up a mobile-emulated browser session for you inside the web app.

Network and Console Logs in Real Time

When a real failure happens, here is where this feature comes to action. You can see console logs and network time in real time while the tests are executing, so if something fails, you have direct access to all the information you would need in order to see why it failed. 

‘Accept Changes’ Feature

One thing that can also happen is that tests fail due to changes that have been planned. For those cases, instead of updating the test manually, you can use the “Accept Changes” feature. It allows you to take that failure as the new expected behavior for that test and the next time you run it, it will look after that change also.

Full Video Recording of the Test

Once the test finishes running, you can see a full video recording. This is a very common feature in other tools, but it adds a lot of value when it comes to debugging and checking if the test ran as expected.

Easy-to-Read Test Plan 

While you’re recording the test, a very easy-to-read test plan is created automatically on the left side of the page. It uses the page title to group the steps you’re performing which makes finding them really easy.

Easy and Accurate Visual Validations

While recording your test, you can add visual validations. They’re quite simple to make and work really great. 

Version Control

With version control, you don’t lose track of previous changes. You can record a test again and still be able to see the previous states and roll back if necessary. 

Available for Mobile and Tablet Resolutions

You can choose between desktop, mobile and tablet resolutions when you create your test which makes this feature perfect for testing responsive pages.

Parallelization

One point that we consider really important is that you’re allowed to run as many tests at the same time as your page supports. Since all the tests start running in a pristine environment on a cloud-based browser, you can have as many instances as you want running at the same time. This makes the execution time for an entire suite really short. All you have to do is create a test group with all those tests and run the whole group.

What We’d Like to See Next from Reflect

While all of the above features and qualities of the tool make it very useful, in the future, we hope to see these features added to the tool:

• Multiple keys
• Cross browser support
• Multiple tabs
• iFrame support – the Reflect team tells me they are coming out with this soon 🙂
• Integration with notification tools
• Parameterization
• Use of spreadsheets

The Verdict

Although this tool is still in its infancy, it has much to offer. My colleagues and I believe that the Reflect team will make several improvements and add more features over time. The features it does have today work extremely well.

Another thing we value about it is that Reflect is very open to feedback and suggestions. In the demo, they mentioned some cases in which their users needed specific features and how they worked with them in order to fulfill those needs. Being able to communicate easily and openly with a tool’s developers is really important for getting the most out of it. 

There were more aspects that I thought were necessary in an automation tool because I’d seen them in others, but while talking with Reflect, I realized that, since their approach is 100% recorder based and they want to be on the highest possible level for the user, they don’t need to include, for example, the option to use javascript code since it’s not something that actual users would do.

Recommended for You

12 Top Test Automation Tools to Try in Your Next Project
The True ROI of Test Automation