Heuristics in API Testing for Quality Software

Discover the best strategies for API testing in this detailed guide. Learn how to enhance your testing strategy by covering aspects such as efficiency, adaptability, and security testing of Application Programming Interfaces (APIs) through the most well-known heuristics and a new heuristic proposed by our team.

In the realm of software development, the API testing process plays a crucial role in ensuring that applications function correctly and integrate seamlessly with other systems.

The testing process is crucial for validating the functionality, reliability, performance, and security of APIs. Within this process, the use of an API testing tool or several API testing tools is invaluable for achieving comprehensive test coverage. Moreover, the adoption of API test automation (also known as API automation testing) can significantly enhance the efficiency of the testing process.

Conducting a comprehensive evaluation of an API’s functionalities can be challenging, especially when deadlines are looming. At this juncture, heuristics for Application Programming Interface testing become a valuable tool.

But what are heuristics? They can be defined as principles or strategies that the testing team employs to effectively analyze an API’s quality. Instead of adhering to rigid, detailed procedures, heuristics offer a more agile and flexible approach. This gives the testing team the ability to quickly identify issues and make informed decisions.

Advantages of Heuristics for API Testing

Efficiency in Problem Detection

Heuristics enable the testing team to quickly identify problematic areas of an API. By employing specific principles and strategies, the testing team can focus their efforts on the most critical areas, saving valuable time and resources.

Flexibility and Adaptability

Heuristics are not rigid rules but flexible strategies. This enables testers to easily adapt to various contexts and specific project requirements. Moreover, heuristics can be adjusted and improved over time, resulting in a more effective and agile testing process.

Comprehensive Coverage

By using specific heuristics, the testing team can ensure they evaluate different aspects of the API, such as response to various request types, and security. This enables comprehensive coverage and helps identify gaps in the API’s functionality.

To effectively conduct API tests in general and also API load tests, it’s essential to understand the various types of API testing, such as load testing, automated testing, functional testing, and security testing. Employing a range of API testing tools can facilitate a thorough testing process, enabling testers to execute API tests across different scenarios and ensure it performs as expected.

Known Heuristics

Here, we present some of the heuristics available that we use daily, to enhance the quality of API testing. Many of these share common points. Nonetheless, each project and context is unique, making it essential to adapt and adjust these heuristics according to specific needs.

1. VADER

Designed by Stuart Ashman, VADER provides a structured framework for evaluating different aspects of an API, from the correct use of HTTP verbs to error management efficiency and responsiveness.

Verbs

This aspect involves assessing whether HTTP verbs are being used appropriately. It’s crucial to understand the business rules of the API you’re evaluating.

HTTP Verbs and their characteristics

Let’s recall the main HTTP verbs and their characteristics:

• POST: Typically used to register new information.
• GET: Used to retrieve information. It can be a listing or a specific entity filtered by some identifier, depending on the product being tested.
• DELETE: Used to delete records, but it’s important to know if the deletion is logical (it becomes inactive) or physical (completely erased).
• PUT: Generally used to update all information of an entity.
• PATCH: Similar to PUT but used to update only a part of an entity.

Authorization or Authentication:

This point refers to verifying the correct implementation of authorization and authentication mechanisms. This includes ensuring that requests are adequately protected and that valid credentials are required to access certain resources.

It’s important to differentiate that authentication is about determining who you are. You can authenticate as long as the system recognizes you are registered. Authorization determines what resources you can access.

Data

Here, the integrity and accuracy of the data managed by the API are examined. The focus here is to ensure that information is stored, processed, and transmitted correctly according to the established business rules, reviewing allowed types or data structures, allowed values, maximums and minimums allowed, and structures.

Errors

This point involves checking how the API handles errors. This means evaluating the HTTP status codes returned in error situations, as well as the clarity and usefulness of the error messages provided.

Understanding the business rules, we can use data sets that we know should cause an error or send malformed requests and evaluate if that error is being appropriately handled, if an error code and a suitable message for that circumstance are returned. For example, returning error codes 500 is a bad practice.

Responsiveness

This last point assesses the API’s speed and efficiency in terms of response time. It ensures that the API is fast enough to meet the system’s performance requirements.

2. POISED

This heuristic was developed by Amber Race and is an acronym representing six key areas to be considered to ensure the system’s quality and robustness.

Parameters

Refers to the evaluation of the parameters of each API operation under test. Essentially, it involves experimenting with the data, passing empty or random parameters, modifying the allowed data types, etc. Moreover, verify that the server responds appropriately.

Output

The goal is to corroborate that the server’s response is as expected. Experimentation can be as extensive as the business rules allow, sending requests whose outputs should be some specific message according to the business rule and validating that it meets the specified requirements.

For example: If registering a user is only allowed if the person is of legal age, sending data of a minor should result in an expected message like: “The user cannot be underage.”

Interoperability

Considers the system’s ability to interoperate with other systems or components. It’s about evaluating if the system can effectively communicate with other systems, services, or devices.

It won’t always be necessary; understanding the API’s purpose, in which other systems it must be integrated, or who will need to use it is vital. In this sense, it’s a good opportunity to evaluate if the documentation is good and understandable for developers who need to integrate a system with our API under test.

Security

Focuses on the system’s security. Besides authentication and authorization, it suggests taking a step further and evaluating other security aspects like data encryption, and identifying potential vulnerabilities. For example: checking if it’s possible to send and store JavaScript code through a POST.

Error Handling

Similar to VADER, here the system’s handling of error situations is examined. Again, we must focus on negative test cases and force all kinds of errors we can think of, such as not sending mandatory fields, playing with boundary values, or using data types not allowed within what the specifications tell us.

It’s important to provide feedback on how the server is communicating the errors if the messages are clear.

Data

This encompasses the data integrity, the way data are stored, retrieved, and processed. For example: if the data stored through a POST are correctly obtained through a GET, as well as, if the deletion of the data should be permanent or is a logical deletion. Accessing databases can be a good idea to complement this type of test.

3. TATTA

Mark Winteringham, the author of the book Testing Web APIs, distinguishes between “Testing the API” and “Testing Through the API”. Both approaches involve evaluating the functioning of a web service, but they focus on different aspects of the testing process.

Testing the API

• Verify that a third-party web service is accessible.
• Check if the web service meets expectations and is operational.
• Evaluate if the web service returns the expected results.

Testing Through the API

• Observe the behavior of the web service.
• Evaluate how the web service processes data and transforms it to return results.
• Verify the correct structure of the records returned by the web service.
• Evaluate how the web service handles situations such as data overflow (e.g., an integer too large).
• Confirm if a redirection occurs after an unauthorized request.

The chosen approach may depend on different factors, such as the tools used and the specific purposes of the test.

4. LHTRAFFIC

Designed by Karol Szewczak, LHTRAFFIC is derived from Left-handle traffic and provides a comprehensive framework for evaluating critical security and functionality aspects in the design, development, and testing of APIs. This approach addresses issues ranging from potential information leaks to injection and data manipulation problems.

Leaky APIs

Refers to APIs that may be revealing information they shouldn’t. It’s important to inspect the data transmitted across the network, assess whether users of the API really need all that information, and if sensitive data that shouldn’t be exposed are being sent. It also involves reviewing response headers to ensure they don’t reveal details about the underlying frameworks.

Hidden APIs

Refers to APIs that the development team didn’t think people would find but are visible in various ways. This involves checking if a “robots.txt” file is provided to prevent crawlers from accessing important resources, avoiding the exposure of administrator endpoints, and reviewing automatic documentation, such as Swagger, to ensure it doesn’t reveal sensitive information or private APIs.

Tampering Requests or Responses

Studies where the validation of the parameters sent with API requests is performed. Warns about the possibility of data manipulation by an intermediate attacker or a proxy. This means that if there’s a web or mobile system integrated with our API, validations cannot only be on the user interface (frontend) side, but our backend must have the necessary defensive code to mitigate possible submissions of unsuitable parameters.

Authorization or Authentication

Focuses on the API’s authorization and authentication. As mentioned here, it’s important to know that with a set of credentials (correct or incorrect), a person can authenticate or not. And if they manage to do so, we need to test what resources they can access.

It’s an opportunity to understand the system well, ensure it’s well documented, and answer different questions. For example: what types of authorization are supported, if it’s possible to bypass them (e.g., not passing a token in requests), how authentication errors are handled, if credentials are sent securely, and if user enumeration can be performed, among other considerations.

Fuzzing

Recommends conducting “fuzz testing” on the API using malformed or unexpected data to detect potential vulnerabilities. This involves testing with different types of data, such as numbers or strings, and using tools like the “Big List of Naughty Strings”. It’s a constantly evolving list of character strings that have a high probability of causing issues when used as user input data.

Forgotten

Explores the possible presence of default headers that might have been accidentally left enabled. Also emphasizes the importance of disabling automatic documentation, like Swagger, in production environments if it was initially intended only for the development environment.

Injections

Refers to the search for possible injections, such as the well-known SQL injections, and evaluates different injection points, including headers, parameters, and the request body.

Content-type

Examines if the Content-type header aligns with the sent MIME type. Also studies how the application responds when a non-supported content type is sent and whether the API supports multiple content types.

ICEOVERMAD

Created by Ash Winter, it not only seeks to contemplate functional aspects of the API but also aspects related to the performance and architecture of the system.

Integration

Aims to evaluate how consumers will integrate with the service, when they will do so, and whether it’s intended to be rendered in a browser, generated in a file, or another consumption method.

Consumers

Seeks to identify who the service consumers will be, whether humans or machines, and understand what problem the service solves for each consumer.

Endpoints

Involves analyzing the form of the endpoint and how it is accessed, whether it’s a single endpoint, multiple endpoints, or if it routes through a load balancer. Evaluates the level of security applied.

Operations

Focuses on identifying the business functions performed by the service, if they can be assigned to current functions performed through a user interface, if the operations have descriptive names and are legible to humans and machines, and if they handle sensitive data.

Volume

Involves evaluating whether the service will be used at a high volume concurrently or sporadically with unique high-value requests, if the size of the individual transactions is a problem, how API sessions are managed, and if the target architecture is clustered.

Error Handling

Involves analyzing how the service handles errors on the server and client side, whether the errors are informative and/or detailed, and how connectivity loss with the database is managed.

RESTful

Consists of evaluating whether the service has the characteristics of a RESTful service and determining if this is desirable in the specific context.

Modularity

Refers to analyzing how the service components are distributed, how they interact, whether they can exist independently, and if they can fail among themselves.

Authentication

Involves investigating how people authenticate within the service, what permissions are applicable, and how that changes the service’s operation, what levels of security are used, and whether data is sent or received encrypted.

Definitions

Concerns determining what defines the service’s inputs and outputs, whether WSDL, WADL, XSD, XSLT, or another method is used, what limits this imposes on the service, and what HTTP methods are used and for what purpose.

BINMEN

This is a heuristic devised by Gwen Diagram and Ash Winter. It is a simple heuristic that focuses on verifying the server’s error handling like other heuristics but breaks down into specific points to induce or force those behaviors.

Boundary

Boundary testing or “Boundary” refers to testing the API with values that are at the limits of what the API is supposed to handle. This might involve testing the API with the maximum and minimum possible values, as well as with values slightly above and below these limits.

Invalid Entries

Invalid Entries consist of tests where entries that should not be accepted by the API are used. They are conducted with the purpose of ensuring that the API properly handles invalid entries, either by returning an error in a controlled manner or by ignoring them.

Nulls

In “NULL,” values are sent to the API to ensure it can properly handle such values. Some APIs may have issues handling null values, which can lead to unexpected errors.

Method

In “Method” testing, the aim is to verify that each of the API’s methods operates according to the documentation, is consistent, and predictable. This can include tests for the methods GET, POST, PUT, DELETE, etc.

Empty

Similar to “NULL” testing, “Empty” tests involve sending empty values to the API. This can help identify issues that may arise when the API receives an empty input.

Negatives

Negative testing involves sending API inputs that are not valid or known to cause an error. The goal of these tests is to ensure that the API can properly handle these errors and not fail or stop functioning.

Wrapping Up

The above heuristics comprehensively cover the critical aspects of API testing. However, each heuristic has its strengths and focus.

We can take the common points from them, combine different relevant points from each, and add other aspects that we consider important based on our experience, to generate a more complete and diverse framework.

Next, we propose another heuristic that encompasses the main aspects on which it is important to emphasize, a summary and an example of what we recommend to consider at each point:

HEURISTIC BADVIPERS

With BAD-VIPERS, testing teams could address functional aspects, security, performance, and aspects related to the integration of the API, and its documentation, thus offering a more holistic and integral view of API testing.

To explain the heuristic step by step and to show some examples, let’s consider a POST operation on a certain endpoint that is responsible for creating a new user in the database, based on the values sent in the body of the message with basic data of a person or a system user:

VIPERS

Verbs

This aspect focuses on assessing whether HTTP verbs are being used appropriately.

Example: Ensuring that a POST to the /api/users endpoint creates a new user, while a GET retrieves a user or a list of users. A common mistake would be to allow user creation with a GET, which is incorrect as it violates RESTful principles.

This aspect refers to evaluating whether HTTP verbs are being used appropriately.

Integration

This aspect evaluates the API’s ability to effectively interact with other systems or components. It involves determining if the API can efficiently communicate with other systems, services, or devices, based on its purpose and user needs.

Example: When integrating /api/users with /api/login, confirm that after creating a user with POST, they can be successfully authenticated through the login endpoint, ensuring both APIs work together coherently.

Parameters

This involves experimenting with data, passing empty or random parameters, modifying the types of allowed data, etc., and verifying the server responds appropriately. You can use techniques such as fuzzing with malformed or unexpected data to detect potential vulnerabilities.

Example: Send a POST request to /api/users with a body like:

{

"username": "",

"email": "[email protected]",

"password": "12345"

}

To check that the server properly handles empty parameters and returns a specific error for the missing username.

Error Handling

API error handling is examined by evaluating HTTP status codes in error situations and the clarity of error messages. This involves inducing errors, such as omitting required fields or using disallowed data, and examining if an appropriate message is returned.

It’s essential to analyze how connectivity loss to the database is managed and how errors are handled on both the client and server sides.

Example: Omit the email field in a POST request to /api/users and expect a 400 Bad Request with an explanatory message like {“error”: “The ’email’ field is required”}.

Responsiveness

An important non-functional aspect of performance is evaluating the API’s speed and efficiency in terms of response time. Ensure the API is fast enough to meet the system’s performance requirements.

Example: Measure the response time for a GET request to /api/users. If it takes more than 500ms to receive a response, you should investigate the endpoint’s efficiency and optimize it. The results may suggest the need for performance testing.

Security

Security in APIs involves exploring private or undocumented APIs and checking for potentially sensitive data in the API. The presence of confidential information in the URL poses significant risks.

Private APIs require security measures to limit their access. It’s crucial to determine the use of SSL to enable secure data transmission.

You should protect the API to mitigate threats such as XSS attacks. In addition to authentication and authorization, it’s essential to evaluate data encryption and detect potential vulnerabilities, such as the possibility of sending JavaScript code through a POST.

Example: Test the /api/users endpoint by sending a POST request with a payload that includes a malicious script in the username field to ensure the API is protected against XSS attacks.

BAD

Behave

It’s important to understand the expected behavior of the API in general terms, understand the business, and devise a test plan prioritizing the most critical aspects.

It’s crucial to pay attention to the server’s responses. Our objective is to confirm that these responses align with what we anticipate. By crafting requests based on business rules, we can perform detailed testing to elicit specific messages from the server. This way, we verify that these messages meet the established requirements.

Example: Make a POST request to create a user on /api/users with correct data, and check that the server returns the expected output, for example, a success message with the ID of the created user.

Authentication

Authentication in API testing focuses on verifying if identification mechanisms are correctly implemented. Requests must be sufficiently protected and require valid credentials for access.

This authentication process allows determining the user’s identity, enabling the system to recognize only registered users. In addition, it’s necessary to analyze and test how authentication errors are handled, how data is transmitted securely, and how user sessions are managed.

Example: Attempt to make a GET request to /api/users/private without including an authentication token to verify that the system correctly rejects the request and returns a 401 Unauthorized.

API testing in terms of authorization focuses on checking the accesses allowed to users. This process is crucial to determine what resources an authenticated user can access. It includes understanding what types of authorization are supported and testing situations where authorization is attempted to be bypassed.

Example: Try to update a user’s information with PUT to /api/users/{id} without the proper permissions to check that the API restricts access and returns a 403 Forbidden.

Documentation

In our role, it’s important to validate the documentation of what we test, and determine if it’s easily accessible, clear, and organized. This review seeks to ensure that the necessary information is efficiently locatable, and coherent and includes examples that help detail the API’s functionality.

Example: Confirm that the documentation for /api/users clearly details how to make requests and what responses to expect, including examples of requests and potential errors.

Conclusion

On the whole, the evolving landscape of the global API testing market demands continuous adaptation and enhancement of testing strategies.

By adopting a mix of these heuristics and leveraging the latest API testing tools, testing teams can boost their APIs to deliver the expected performance, security, and reliability, thus meeting the high standards required in today’s digital world.

Empower your software with expert API testing! Contact us to discuss how we can help you grow your business.

Follow us on Linkedin & X to be part of our community!