An introduction to this popular crowdsourced testing platform

Adding to our collection of software testing tool reviews, we wanted to give our take on one of the tools we’re asked most about from clients and prospects: Rainforest QA.

We often hear, “Can I just rely on Rainforest QA instead of ramping up my test team? Is it enough? Is it reliable?”

As a testing services company, we actually use it in some of our client projects, helping them manage their crowdsourced test efforts. So, in this post, we’ll share our experiences with it and our best recommendations.

What is Rainforest QA?

Rainforest QA is an on-demand, crowdsourced software testing platform boasting 50,000 testers all around the world. API-driven, the platform is accessed via the web and has an open-source CLI app that lets you easily connect it with your CI tool (Circle, Jenkins, Bamboo, Travis, etc.). Rainforest manages and grants testers access to testing infrastructure (web and mobile devices, virtual machines, etc.). What is neat is that you see test reports being generated just minutes after you input your test scripts which can be written in plain English (no need to learn a new language). When you execute a suite of tests, Rainforest QA assigns three testers per test per browser or platform to test in parallel.

What to Know Before You Start Crowdsourcing

 

Like any powerful tool, it does require a bit of patience when learning how to use it and to optimize its output.

For example, it’s important to think about how to set up a repeatable testing environment and how to structure tests so that they are understandable to someone who isn’t familiar with your system. Think of it as an extra exercise in analyzing your user experience… If it’s difficult to write easy-to-reproduce test cases, this is probably symptomatic of a system is not that easy to use!

There are also some challenges of crowdsourced testing that need to be managed from the start. The benefit of this kind of testing is that you get a lot of mostly accurate feedback very quickly, and it’s highly affordable to boot.

But, it’s also fundamentally driven by folks who are not always professional testers and each may have very different levels of experience, so the quality of their work may vary wildly. Though it isn’t an outright negative, it’s a cost to keep in mind as you weigh all the benefits of crowdsourced testing.

Benefits of Rainforest QA

One of Rainforest QA’s best attributes is its user-friendly interface which offers handy functionalities like being able to tag test cases to group them later.

Personally, it has been immensely useful in projects to see a test suite being run immediately, with results coming through while it’s still in progress. In one of our projects, the regression testing suite contained 240 test cases (each with five steps on average) and it only took 80 minutes for it to be executed 100%.

So, as you can see, Rainforest is very easy to get up and running. You can go from nothing to running tests in a matter of minutes.

Rainforest QA can best be thought of as a UI-driven functional system test automation service.

The fact that real humans are executing the test scripts is a benefit in that it allows for easy detection of unanticipated errors or display issues.

Because tests can run in parallel, a full test run takes no more than the duration of the longest manual run in a set, which itself may not take significantly longer than a custom-built Selenium run. Creating and maintaining test scripts is however vastly easier (it can be done without any coding skills) and it’s also possible to create them during development.

Cross-browser testing is enabled with a simple checkbox on a test and older browser versions are supported.

Besides all of the above, these are some of the key benefits of Rainforest QA from my experience:

• It gives you the ability to access many more devices (both Android and iOS) and platforms (Windows and macOS) than you are typically able to access in-house.
• It’s easy to write tests and embed steps (in order to reuse them and foster maintainability).
• It’s easy to review the results in real time.
• It’s easy to verify failures with a mostly clear explanation provided by the testers around the world and with the help of the screenshots displayed in every step.
• The client support team is quick to respond.
• You can include a dependency reference when editing or removing test cases.
• You can shift from a sequential to a “parallel” testing process to keep pace with development, and in turn, reduce testing time and get feedback earlier during the sprint

How to Use Rainforest QA in Agile Contexts

You might be thinking that this tool seems suitable only for waterfall projects, to do some acceptance testing at the end of the development.

But, at Abstracta we’ve managed to get Rainforest QA running in some of our clients’ Scrum teams, often with two-week sprints, quite successfully.

Here’s how.

During the sprint planning meeting, the developers, testers and the Product Owner all agree on what work is to be done in the current sprint. It’s also a point when everyone comes to a mutual understanding of the requirements and the definition of the user stories.

While developers are coding, testers finish designing and writing the tests on the Rainforest QA platform. Once a story is released, a tester coordinates the story’s test case execution in its specific environment, obtaining early feedback.

Finally, at the end of the sprint, a full regression can be made on the release candidate by executing all the critical regression tests plus the tests of the new functionalities. The most relevant test cases designed for the new stories are also included in the regression test suite for the following releases.

Recommendations for Using Rainforest QA

If you’ve read this far into the post, it should be pretty clear that we would recommend adding Rainforest QA to your testing tool belt! Here are some best practices and nuances of crowdsourced testing to keep in mind.

Firstly, sometimes testers follow the test steps literally—exactly as they are written—meaning an ill-placed word can result in a failed test when the functionality is actually completely fine, so be careful with this.

There can often be false positives (meaning, a bug or exception is reported when there isn’t one) due to a tester getting confused by the test directions. For example, if you ask the tester, “Do you see a page that says ‘Welcome!’?” and the tester sees a page on the website that says “Welcome” but without an exclamation sign, the tester may think something is wrong, and the test step will fail.

This problem is only exacerbated in longer tests, thus it’s important to make the directions as simple and clear as possible, keeping the test cases short.

Another thing to keep in mind is that multiple testers (e.g., three) will hit the same system’s feature simultaneously, so whatever setup and changes that one tester does mustn’t interfere with what another tester does. This can be figured out, but it will require some special design for the tests.

Last but not least, if you have 240 test cases like we did, which get multiplied by three testers each, you will end up with 720 people hitting your servers. Is your testing environment ready for this? Before writing loads of test cases, make sure your system and environment are ready for the multitudes of testers that will bombard it.

Rainforest QA to Supplement, Not Replace Your Testing Team

Rainforest QA is a great tool to expand your test coverage and augment the capacity of your testing team, but it is best paired with an internal tester or test team.

The internal (which may include outsourced) testers are the ones who will take the time to manage the crowdsourced testing, using their understanding of the business for ensuring best outcomes, writing the tests cases, coordinating test runs and analyzing the test reports (considering the business priorities, analyzing impact, risks, etc).

While it may take some effort up-front, what we have found is that:

 

What do you think? Have you used Rainforest QA? How do you integrate it with your workflow?

 

---

Recommended for You

How to Get Started with Outsourced Software Testing
Selenium vs Watir