Model-Based Testing Using State Machines

What is a state machine and how can you use one to improve your testing strategy?

When testing software systems, it’s very important to build a mental model of what you understand about the expected behavior, the interactions between the user and the SUT (system under test), the conditions and the different situations that will help to determine its correctness. There are certain mechanisms to draw the model on paper – one of which is a state machine.

In this post, I’ll cover model-based testing using a state machine and how it can help improve your testing.

State machines are models for the behavior of a system or a part of a system. They’re represented with graphs where the nodes correspond to the states and arrows to the transitions between said states.

The image above shows a super simple and graphic example that models the behavior of a light bulb which can have one of two states at any time: on or off. The arrows show the transitions between these states which are associated with an action (pressing the light switch).

These models are extremely useful in the area of ​​software development to abstractly model the behavior of a system, being able to understand it better and discuss with others what it’s expected to do and what it isn’t.

From a testing point of view, state machines can be super useful, although unfortunately, they’re quite underutilized in the industry. 

In light of that, here is what I’ll share specifically in this post to help you get started:

  • A simple example of a state machine
  • A more complex, real case scenario
  • A tool that will help you apply this testing technique

A Simple Model

Let’s continue with the example of the light bulb. We can represent this everyday situation with the following state machine:

model-based testing light bulb example

Let’s see in more detail how this model is composed:

  • Status: The states are conditions in which a system is while waiting for one or more events (switch on, off, etc.). The same event can cause different behaviors for the different states (this distinguishes the states). For example, the same “press the switch” event generates different actions depending on the state in which the system is located. 
  • Event: The “external” stimulus that the system receives that causes a change in its state. For example, the “press the switch” event causes the light to change its state. Once an event occurs, the system can change its state, remain in the current state and / or execute an action. 
  • Action: An action is an operation that starts from an event. For example, after “pressing the switch” the “turns off” action is executed.
  • Transition: A transition is represented by an arrow indicating that there is a change, a passage from one state to another. In each of these arrows, we can see concepts of guards (conditions), events and actions. 
  • Guard: A condition that must be met for the event to cause it to go through a certain transition. In the example, we could add one more transition (another arrow) that goes from “off” to “off” (a loop to the same state) by adding the guard that says “there is no electricity”. 
state machine example for light bulb

Now, let’s see how to generate test cases for state machines according to the coverage criteria.

State Machine Coverage

State machines can be used to design test cases that cover a system according to some coverage criteria. Some of these coverage criteria are: 

  • State Coverage: This criterion is satisfied when the test cases cover all the states
  • Transition Coverage: All transitions are traversed by the test cases
  • Transition Pair Coverage: For each state, all combinations of input and output transitions are covered

This type of technique is one of the most used in the world of Model-based Testing (MBT), for which I recommend checking out Harry Robinson’s work.

Following the example of the light bulb, let’s see what happens with these test cases: 

  • Test case 1: The light bulb is off and without electricity, I press the switch. 
  • Test case 2: The light bulb is on and without electricity, I press the switch. 
  • Test case 3: The light bulb is on, I press the switch. 

It is interesting to note that just with test case 2, you already have state coverage. If you wanted to cover all the transitions, then you need test cases 1, 2 and 3. In order to cover the pairs of transitions, you should make even more flows over the system. This shows how one criterion subsumes another.

A Realistic Scenario for Testing State Machines

Now, let’s go to an example that’s a little more complex or more similar to what you’d face when testing a system. 

In this example, you’ll see that we’re not tied down to a single representation of state machines and this technique could be applied manually.

Imagine you’re testing a system that manages inventory items. The objective is to design the test cases for the creation of items, taking into account the entire item’s life cycle and the different people who have to perform different actions and validations on the information uploaded to the system. 

This is a great example of a scenario in which you could model the behavior of the items with a state machine and derive the test cases from it. The items may have different states and the expected behavior for each action or functionality of the system will depend on the item’s state.

The following figure presents a partial state machine for the concept, item. To the left of the figure you can see the actor involved in the task and, in this specific case, the guards are represented as decision points in the diagram. 

Inventory management system model-based testing example

The states determine different behaviors of the instances of items. The transitions correspond to system functionalities that cause the item to change its behavior, its state. 

The state machine shows the expected behavior of the entity, item, in its life cycle. What will be tested is to cover all the transitions and nodes of this diagram. The result is a set of sequences to follow about the state machine that defines the interesting test cases. For this state machine, they would be the following:

  • Case 1: Preliminary item entry, verify data and refuse purchase.
  • Case 2: Preliminary item entry, verify data and approve by the buyer responsible for the area.
  • Case 3: Preliminary item entry, verify data and approve by the buyer responsible for the area, complete the data by trade marketing, the purchasing department manager does not approve the purchase because it is a non-promotional and saturated item.
  • Case 4: Preliminary item entry, verify data and approve by the buyer responsible for the area, complete the data by trade marketing, the purchasing department manager does not approve the purchase for a promotional and saturated item.
  • Case 5: Preliminary item entry, verify data and approve by the buyer responsible for the area, complete the data by trade marketing, the purchasing department manager approves the purchase for a non-promotional and unsaturated item.
  • Case 6: Preliminary item entry, verify data and approve by the buyer responsible for the area, complete the data by trade marketing, the purchasing department manager approves the purchase for a promotional and saturated item.

So far, these are high-level test cases (abstract test cases). That is, there are only the functional sequences to be invoked and the conditions on the data. Then, you must define the specific data that allows you to run the tests on the system.

At the time of execution, the state machine is used as an oracle (to determine if the result is valid or invalid), since it must be verified in each step whether the expected state was reached or not. Each abstract test case may correspond to one or more specific test cases based on the different data selected.

So, from this example, we’ve seen how you can model the expected state of a system with a graph and hence, derive test cases, taking into account the coverage criteria you want to achieve. 

Testing State Machines with GraphWalker

Although this testing can be done manually, there are tools that facilitate the generation of test cases from a state machine model, ensuring the desired level of coverage. One such tool is GraphWalker which we’ll go over and look at some of its benefits.

GraphWalker allows you to design a state machine in a graphic format and automatically generate test cases to cover the model based on different criteria. 

GraphWalker has an editor where you can create the model, like the one I made earlier of the light bulb, for example. To use this editor, I recommend you check the GraphWalker user manual because, as of today, although it’s not too complex, it’s not too intuitive either. 

Once you generate the model, download it in the JSON format from which the GraphWalker application will generate a series of paths according to the coverage criteria chosen. 

For all options and details, I recommend reviewing the product documentation. In addition to what we saw here, the tool offers many more possibilities and it’s also useful for automating the test execution, integrating with tools like Selenium. 

As an example, the following is an output of the program applied to the light bulb state machine, where you can see the sequence of steps that it suggests to follow in order to cover all the transitions. 

  • {“CurrentElementName”: “Off”}
  • {“currentElementName”: “press switch / turns on light”}
  • {“currentElementName”: “On”}
  • {{“currentElementName”: “[there is electricity] press switch / turns off light”}
  • {” currentElementName “:” Off “}
  • {” currentElementName “:” [no electricity] press switch / no effect “}
  • {” currentElementName “:” Off “}

Different levels of coverage can guarantee us more or less exhaustiveness. Then, to decide what coverage to use in each case, consider the criticality of the functionality tested and the amount of time you have available for testing.

More About GraphWalker

This GIF below shows in a very simple way, the concept of going through a state machine, which can be used for “manual” testing as well as for automated testing. 

What’s interesting about GraphWalker is that it allows you to have a bridge between the model and the test code, with which we can “execute” the model and thus execute automated tests.

To understand more about this concept, I recommend listening to talks by Kristian Karl from Spotify, there are several that he has given about this, and here is just one of them.

Incorporate State Machine Testing for Better Test Coverage

I hope after reading this, you feel motivated to use this useful and powerful technique and by doing so, help professionalize more and more the work that we do in testing.

I’d love to encourage you to try this model-based testing technique and GraphWalker. If you dare to share examples, repositories where you have been testing the tool, questions or comments, I’d love to hear from you… leave a comment below! 

Recommended for You

Improve Your Software Testing Strategy: A Software Testing Maturity Model
How Can You Optimize the Cost of Software Testing?

163 / 435