Lessons learned from helping organizations foment a DevOps culture through Agile testing practices

At Abstracta we work with many companies, several of whom already have a DevOps culture and others whom we’ve helped to define and promote it. Over the years, we’ve seen DevOps gaining popularity, and most teams are on their way there. In this post, I’ll share some lessons from our experiences in helping companies with their agile transformations.

From what we’ve seen, we’re convinced that teams with a DevOps culture work better, obtain better results and are, just… happier.

This illustration above by Dan Ashby seemed great for illustrating the concept, showing how a tester is involved in DevOps.

As you can see… we are everywhere!

In DevOps, testing should take place in each of the stages of the development process. DevOps proposes (as well as many agile methodologies) that testing is NOT something that is only done at the end, but is done continuously, associated with each task.

So, keep in mind that in this post, all of the ways we point out are necessary for adopting a DevOps culture are also part of adopting an Agile culture as well.

Continuous Feedback

Let’s start with “continuous feedback” since generating a DevOps culture implies generating a “continuous feedback culture,” in which everyone learns from the others on their own team as well as the teams with which they collaborate.

The first step we always suggest to take when promoting this cultural change within an organization is to have retrospective meetings. Yes, the famous retros that come from the Scrum methodology. It’s a ceremony where the focus is on getting feedback about the process and the way in which the team is working. In Scrum, these meetings are held frequently, and that’s why it helps to implement the practice of giving and receiving feedback.

Testers naturally tend to look for opportunities for improvement and provide feedback, which makes us fit for “testing the process.”

Often times, teams brush off the idea of holding retros as a waste of time, initially. But, after giving them a real try, they can’t imagine not having them. We often hear, “How did we manage our work before? How could we not do this kind of analysis?”

Thanks to instilling continuous feedback by holding retros, learnings are shared and the team has a space to experiment and innovate. In addition, everyone is aware of any problems that were had, encouraging transparency, visibility, and inspection, in order to adapt the process and improve it.

It’s best to start with carrying out simple group dynamics (what went right or wrong, what to start and what to stop doing). Then organize the discussion, prioritizing the topics to be discussed. It’s essential to focus on an action plan so that the team can then follow up on the proposed ideas. As time goes on, you can try different dynamics to spice up retros and make them more entertaining, while thinking about problems from a different perspective.

To point to a DevOps culture, what is also important here is who attends these meetings. If they’re only held amongst the developers, then it isn’t truly a DevOps culture. At least someone from testing, management, business, support, and infrastructure, should be represented. In our experiences, ONLY by including everyone have we seen true unity within the team and a strong culture change take place.

Planning

The first thing we have to understand is that when talking about agile methodologies, the paradigm of estimation and planning shifts. Instead of setting the scope and estimating the time and budget, here we set the time and budget, and estimate the scope.

As there is no fixed scope, the team won’t have everything documented nor will it fully establish what will be accomplished, definitively. The team will see what can be delivered, and go into more detail as the development progresses.

Testers are fundamental in the definition and refinement of the software’s requirements. In fact, the idea is that the tester is involved starting at the same time as the developer, or even sooner. In DevOps, not only the tester should be involved from the start, but also someone from infrastructure, support, etc. This dramatically changes the results, as when discussions take place about what is going to be implemented, the team also considers how it will be tested, operated, put into production, what’s the anticipated user feedback, etc.

There is another paradigm shift that greatly impacts results, which is the participation of the whole team during the planning and estimation. Amongst everyone it is decided how complex something is to build, and if it’s possible to do in the sprint or not. Then, at the time of assigning tasks, everyone assigns themselves something, assuming responsibility for it. In a way, managers lose some power, but the benefit is that the team is more committed since the work is not something imposed upon them, but rather something they have decided to own.

When these changes take place, the team becomes more predictable. Its ability to aim at a target improves and it fine-tunes its ability to estimate what can be done in a certain period of time.

Agile Builds

The build stage in the DevOps culture is defined by a high level of agility, teamwork (by its truest definition), short iterations, and continuous feedback. In this stage, product increments are frequently sent to the customer, so it’s important to pay attention to how they’re efficiently integrated at low risk. This is when automation comes into play, either in testing, deployment, or continuous integration, etc. For some clients we have worked with, this is the point at which we’ve found some barriers that hinder them from reaching this point of agility.

One such barrier commonly occurs when there is specialization in the modules, where only one developer is able to work on a specific module and doesn’t know the details of the rest. There is no knowledge sharing, and it becomes very complex to support the development of a module when someone goes on leave or for some reason is unable to continue working, and so, progress halts or becomes very expensive.

Pair Programming

To combat this issue, we have suggested to our clients to incorporate pair programming. The person who has the greatest responsibility for the development of a module can be said to be the “driver,” but now they’ll also have another team member, a “navigator,” who reviews the developed code and supports them to understand the logic and look for any best practices. At the same time, each one plays the opposite role in other modules (either the driver or the navigator player or the substitute). This practice helps to ensure that there will always be someone who can take responsibility for a module, limiting the dependence on a single person to fix any problems or continue development.

The goal is to link everyone across the different modules over time, until each developer can support or develop any given one and open it up so that everyone can take on any task within the sprint backlog, thus making the team more agile.

Test Early

Another typical problem is having tests that act only as a filter before going into production, which causes a cascading effect within the sprint and an overloaded tester at the end. A bottleneck occurs in the process before the production goes on, and testing (or worse, the tester) is left unable to finish on time.

A fundamental aspect to prevent this is to think about tests before the development, that is, as the requirements are being defined, the tester starts writing the acceptance criteria, test ideas, or even test cases themselves. Also, testing’s support in test automation and in CI/CD approaches is critical for working in the best way.

Kanban

Apart from the previous areas we’ve covered so far, something that we’ve seen firsthand provide excellent results is following another agile methodology, Kanban.

You may already be familiar with Kanban’s use of boards, where individuals organize their work in different columns, and thus the whole team has visibility into who is working on what.

Image Source: Olga Colino

On a board with columns that are defined for a certain project, tasks or stories will pass through the columns, changing state. If a developer starts working on developing features, and as they finish, they pass them to the “testing” column, regardless of whether there are several accumulating there already, a false sense of progress may arise. The developer will think that he or she is making progress, but in reality, their work won’t be put into production right away because testing is the bottleneck. Or worse, the tester will be seen as the bottleneck (Again!). What can a team do to avoid this?

What the Kanban methodology proposes is to limit the work in progress (WIP). Imagine that a team sets the limit in the testing column to three, and there are already three tasks in that column. The developer’s task will then remain “In progress” and can’t move forward until a space becomes available in the “In testing” column. To speed up this release, the natural solution will be for the developer to help the tester with the other three tasks, either helping to test or by brewing the tester some coffee!

Another thing that should be done in these situations is to analyze the reason why tasks accumulate in the testing column. Maybe after doing some analysis, the team notices the fact that the features that arrive at this stage are still very “green.” Something we’ve done to help with this (in more than one project) was to have the devs and testers agree that the devs would do some basic testing first (for example based on a checklist). This leads to the tasks being passed back and forth between testing and development less often, and the time the tasks spend in the “In testing” column reduces.  

In this way, the whole team is responsible for quality and for things to be released on time. The same idea could be applied to different columns, and if we are DevOps, maybe we also manage this Kanban board with a column “To Deploy”. The best results are given when there is collaboration at this level, having the whole team (devs, testers, ops people) working together to consider something finished when it is deployed.

CI/CD and Deployments

In DevOps, we are frequently delivering new increments that need to be integrated correctly and must verify that all aspects of the software continue to work as expected. This is all expected to occur while making sure no friction is generated between development and operations. CI/CD tells us how to put those increments into production without inflating the cost.

The stages of Deploy (putting into production) and Operations complete this loop that is truly infinite. It would be a mistake to see and treat any of these stages as final. In reality, the operations team must be, as the test team, present from the beginning.

The focus of operations in a DevOps environment is maintained by making builds happen more reliably, more frequently and with some attention towards good automation of deploys. Operations can also play a role in monitoring, as well as some of the more complex aspects of configuration management.

But many times there is physical separation like different floors, different offices or even different companies taking care of the operations. This already implies a barrier to proper functioning, so one of the main things to achieve is the bringing of both parties to the team (and not just physically).

DevOps for Higher Quality Software

When testing helps drive the shift to DevOps, the quality of releases to production noticeably improves, stress is reduced, and risk is better controlled.

In one case where we helped a company shift to DevOps since its testing activities were added in the process as part of the “Definition of Done,” for the first time ever, a new release to production didn’t cause any user complaints at all. Before having a DevOps culture, the team was used to planning one extra week between sprints, after each release to production, for making corrections based on user complaints.

Clearly, this is not something that can be changed from one day to the next, but the results are worth it. Not only to have a more agile culture, which is fashionable but because better workgroups can be created that are more collaborative and improve continuously, naturally bringing about better results.

I hope that after reading this post, you can see the possibility of testing not only driving teams to go further but also helping them take off and reach places they’d never imagined.

Watch our webinar recording with Testim.io, “8 Mistakes Companies Make When Transitioning to CI/CD”:

Recommended for You

Why So Much Talk Around DevOps Culture?
We Want our DevOps Venn Diagram Circle Back – Plus One!