Environments and Infrastructure
Presumably, something everyone can agree on is that there are at least three different
environments to work within: one for development, where the code is compiled and tested locally upon
each modification, one for testers to test the "closed" versions that are ready to test, and the
production environment, where the system code that users face is found.
It never occurs to anyone to go from the development environment straight to the production
environment, right? Because that would be plain crazy…
Unfortunately, some teams do just that, run tests in the wrong environment. There are many that don’t
have even the three basic environments to carry out these practices, which causes many consequences.
For example, this is when one might hear the excuse from developers, "Well, it works on my computer,"
when something isn’t working for the user (when it matters most).
To really test software before it reaches the user, it is essential to have a specific environment for
it, and, sometimes even that is not enough. Especially for automated checks, it is essential to run
these tests with the data for them in isolation.
Furthermore, it’s advisable to have a
separate environment to ensure that they will not be modified (if they are modified, there is a risk of
obtaining false positives).
It’s fundamental to manage the test environment well, considering many of its different elements:
- The source files and
executables of the application
- The test devices and the data
- In turn, the data are related
to the database system in each environment, so it’s necessary to manage the schema and data
from the corresponding database
If there are different tests to be performed with different configurations, parameterizations, etc.,
more than one test environment will be needed, or an environment and many database backups, one for
each set of tests. This means that one has to perform specific maintenance for each backup (for
instance, whenever there are changes in the system in which the database is modified, one must hit each
backup with these changes).
But if one of these elements is not in tune with the rest, the tests will probably fail and lead to
inefficiency. It’s crucial to be sure that each time a test reports an error, it is because there
really is a bug instead of it producing a false alarm.
To solve all these problems at the environment level, there are typically special roles within a team.
An immensely popular industry trend now is to have DevOps. There is a slight controversy over
the term because the original concept was related to the culture of
a team, and not designated to individual roles. But, today you will
see companies hiring a “DevOps Engineer.” According to Puppet Labs, teams that employ DevOps deliver 30
times more frequently, have 60 times fewer failures and recover 160 times faster.
Basically, the role of DevOps is to combine the vision of operations management (infrastructure and
environment), development, and business. Connecting these areas makes it possible to quickly and
efficiently resolve all the needs of the team while placing the focus on the customer and the end user.
As for tools, DevOps usually takes advantage of the facilities of virtual machines as well as other more
advanced ones like Docker and Kubernetes. When it comes to the variety of mobile devices needed
for testing, there are several solutions for avoiding buying and maintaining dozens of devices every
month, from device laboratories to
shared solutions in the Cloud that offer access to devices on a pay as you go basis.
Thus, having multiple, properly managed environments helps avoid questions like, "Are we sure we're
testing on the latest version?” It is also fundamental to have all the environments that testers need,
which may include providing access to different operating systems with different configurations,
different browser versions, or even having access to various mobile devices. Due to the vast array of
devices on the market and the fragmentation within Android and nowadays even with iOS, it’s necessary
to test on as many of the most relevant devices as possible.
Concerning mobile device coverage, a strategy that Abstracta has seen great results with is
"cross-coverage over time.” Whereas testing on a variety of devices requires more runtime, the
cross-coverage strategy aims to improve coverage over time. This strategy simply proposes organizing
executions in such a way so that they are not all ran in each execution cycle, but rather ran
alternately, improving coverage after many cycles. The following images exemplify this strategy:
Whereupon, in the third execution cycle full coverage is reached:
This does not guarantee 100% coverage in each run, but by managing to toggle the test runs,
greater coverage is successively achieved.
This same strategy applies to web browsers, parameters, or many more variables with which testers have
To achieve testing maturity, the necessary environments for test execution are required to ensure that
there aren’t any additional problems besides those that one hopes to uncover through testing.