03. Environments and Infrastructure

Presumably, something everyone can agree on is that there are at least three different environments to work within: one for development, where the code is compiled and tested locally upon each modification, one for testers to test the "closed" versions that are ready to test, and the production environment, where the system code that users face is found.

It never occurs to anyone to go from the development environment straight to the production environment, right? Because that would be plain crazy…

Unfortunately, some teams do just that, run tests in the wrong environment. There are many that don’t have even the three basic environments to carry out these practices, which causes many consequences. For example, this is when one might hear the excuse from developers, "Well, it works on my computer," when something isn’t working for the user (when it matters most).

To really test software before it reaches the user, it is essential to have a specific environment for it, and, sometimes even that is not enough. Especially for automated checks, it is essential to run these tests with the data for them in isolation.

Furthermore, it’s advisable to have a separate environment to ensure that they will not be modified (if they are modified, there is a risk of obtaining false positives).

It’s fundamental to manage the test environment well, considering many of its different elements:

• The source files and executables of the application
• The test devices and the data they use
• In turn, the data are related to the database system in each environment, so it’s necessary to manage the schema and data from the corresponding database

If there are different tests to be performed with different configurations, parameterizations, etc., more than one test environment will be needed, or an environment and many database backups, one for each set of tests. This means that one has to perform specific maintenance for each backup (for instance, whenever there are changes in the system in which the database is modified, one must hit each backup with these changes).

But if one of these elements is not in tune with the rest, the tests will probably fail and lead to inefficiency. It’s crucial to be sure that each time a test reports an error, it is because there really is a bug instead of it producing a false alarm.

To solve all these problems at the environment level, there are typically special roles within a team. An immensely popular industry trend now is to have DevOps. There is a slight controversy over the term because the original concept was related to the culture of a team, and not designated to individual roles. But, today you will see companies hiring a “DevOps Engineer.” According to Puppet Labs, teams that employ DevOps deliver 30 times more frequently, have 60 times fewer failures and recover 160 times faster.

Basically, the role of DevOps is to combine the vision of operations management (infrastructure and environment), development, and business. Connecting these areas makes it possible to quickly and efficiently resolve all the needs of the team while placing the focus on the customer and the end user.

As for tools, DevOps usually takes advantage of the facilities of virtual machines as well as other more advanced ones like Docker and Kubernetes. When it comes to the variety of mobile devices needed for testing, there are several solutions for avoiding buying and maintaining dozens of devices every month, from device laboratories to shared solutions in the Cloud that offer access to devices on a pay as you go basis.

Thus, having multiple, properly managed environments helps avoid questions like, "Are we sure we're testing on the latest version?” It is also fundamental to have all the environments that testers need, which may include providing access to different operating systems with different configurations, different browser versions, or even having access to various mobile devices. Due to the vast array of devices on the market and the fragmentation within Android and nowadays even with iOS, it’s necessary to test on as many of the most relevant devices as possible.

Concerning mobile device coverage, a strategy that Abstracta has seen great results with is "cross-coverage over time.” Whereas testing on a variety of devices requires more runtime, the cross-coverage strategy aims to improve coverage over time. This strategy simply proposes organizing executions in such a way so that they are not all ran in each execution cycle, but rather ran alternately, improving coverage after many cycles. The following images exemplify this strategy:

Whereupon, in the third execution cycle full coverage is reached:

This does not guarantee 100% coverage in each run, but by managing to toggle the test runs, greater coverage is successively achieved.

This same strategy applies to web browsers, parameters, or many more variables with which testers have to “play”.

To achieve testing maturity, the necessary environments for test execution are required to ensure that there aren’t any additional problems besides those that one hopes to uncover through testing.