Automated Accessibility Testing: Comparing axe + WDIO and Pa11y-ci

Let’s dive into accessibility testing! We will explore efficient and effective tools that enhance website accessibility. Our goal is to evaluate their capabilities and coverage based on accessibility standards and learn what the best tool for which task is. Get insights and strategies for incorporating these tools into your web development projects!

Accessibility is an increasingly vital component of web development. It helps to minimize technological barriers and make websites and applications user-friendly and inclusive for all.

However, analyzing an entire website for accessibility issues can be a time-consuming and daunting task. That’s why we want to help you explore some tools. We will delve into two of the most popular options that can significantly assist you in enhancing your website’s accessibility.

We’ll put Pa11y-ci and axe + WDIO under the microscope to evaluate their effectiveness in conducting automated accessibility testing. 

Our goal is to assess the capabilities and coverage of each tool concerning accessibility standards to deepen the accessibility testing. How will we achieve this? By examining a previous Proof of Concept (PoC) with axe + WDIO and undertaking a new PoC with Pa11y-ci.

We’ll also offer insights into their overall performance and suggest strategies for incorporating these tools into web development projects.

Overview – Pa11y-ci vs. axe + WDIO

Pa11y-ci and axe + WDIO are both tools designed to help you validate web accessibility. They do this by utilizing a set of rules based on the Web Content Accessibility Guidelines (WCAG). 

These tools analyze the HTML code of your website and provide you with a detailed report of the findings. This can have various formats such as JSON or HTML. As a result, the team is able to identify and fix accessibility issues. 

To better understand what each tool brings to the table, we’ll go through a brief overview of both Pa11y-ci and axe + WDIO.

Pa11y-ci

Pa11y-ci is an open-source command-line interface that makes it easy to run accessibility tests on your websites.

Some of Its Key Features Include:

✅Automated Accessibility Testing.
✅Supports WAI-ARIA, WCAG 2.0, and Section 508 standards.
✅Easy integration with CI/CD pipelines.
✅Ability to test multiple URLs concurrently.

Below, you’ll find a visual representation of the Pa11y report. This image shows how the report provides a straightforward representation of the conducted accessibility tests. Please note that the report does not offer a comprehensive overview or detailed grouping of the identified issues. Instead, it serves as a convenient way to quickly assess the accessibility status of a site, highlighting the errors that were found.

axe + WDIO

axe is an open-source library for automated testing of web page accessibility. Meanwhile, WebdriverIO (WDIO) is an automation framework for Node.js that provides capabilities for WebDriver protocol implementations like Selenium.

Together, they offer a glimpse into a more powerful automation experience that takes accessibility into account. But here’s the thing: automation alone won’t cut it. We can’t just rely on it and brush aside the importance of manual testing.

The setup process of the framework was not primarily complex due to the integration with axe. This is as simple as importing the “@axe-core/webdriverio” library. Instead, the challenge lies in installing a separate reporting tool to export the results to an HTML report.

Moreover, when utilizing a sitemap similar to Pa11y’s approach, it was necessary to employ Axios to fetch the URLs within the sitemap. Subsequently, a function had to be created to iterate the tests for each page found in the sitemap.

The Perks of Using axe + WDIO Include:

✅Flexible Integration with WebdriverIO Test Framework.
✅Supports WAI-ARIA, WCAG 2.0, and Section 508 standards.
✅Detailed Reporting and Highlighting of Issues.
✅Ability to Test Dynamic and Complex Web Applications.

In the following, you’ll find several images of the axe + WDIO report. These images provide an overview of the results, followed by a list of accessibility issues found in the web application.

Additionally, the report includes sections that showcase results that pass the accessibility guidelines checks. And also results with incomplete and inapplicable accessibility results.

Automated Accessibility Testing: Comparison 

Below, we analyze the two tools based on the outcomes of conducting Proof of Concepts (PoCs).

axe + WDIO PoC

✅On average a 57% coverage of WCAG accessibility issues.
✅Clearer reports, grouped issues, and linked solutions enhance result interpretation.
✅Enabled seamless integration with the WebDriver framework.
✅Provided a flexible test environment
❌More complex configuration setup.

Pa11y-ci PoC

✅Simplified execution through the command-line interface.
✅Allowed testing multiple URLs quickly. 
❌Limited capabilities compared to axe + WDIO.
❌Less precise in detecting all issues.

In a Nutshell – What is the Best Tool for Which Task?

To sum up, both Pa11y-ci and axe + WDIO provide significant benefits for web automated accessibility testing. While Pa11y-ci stands out for its simplicity and speed, axe + WDIO offers more extensive capabilities.

To support manual testing efforts, we recommend utilizing axe + WDIO as the primary method due to its comprehensive, categorized reports. This enables the prioritization of issues and the identification of areas meeting standards or requiring further review.

For CI/CD integration, Pa11y-ci is the superior choice as it is easy to set up and operates swiftly. It can be seamlessly integrated into the Build and Deploy stages to maintain a continuous emphasis on accessibility.

Are you interested in learning more about the suggested strategy for automated accessibility testing and implementing axe + WDIO?  Check out our proof of concept on GitHub. If you have any questions or need assistance, don’t hesitate to reach out!

Looking for support in integrating Pa11y-ci into your CI/CD pipeline?
Contact us! Let’s discuss how we can help you maintain a continuous focus on accessibility throughout your project’s development process.

Follow us on Linkedin & Twitter to be part of our community!