{"id":15870,"date":"2023-07-19T16:33:10","date_gmt":"2023-07-19T16:33:10","guid":{"rendered":"http:\/\/abstracta.us\/blog\/?p=15870"},"modified":"2025-05-05T21:19:21","modified_gmt":"2025-05-05T21:19:21","slug":"automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/test-automation\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/","title":{"rendered":"Automated Accessibility Testing: Comparing axe + WDIO and Pa11y-ci"},"content":{"rendered":"\n

Let’s dive into accessibility testing! We will explore efficient and effective tools that enhance website accessibility. Our goal is to evaluate their capabilities and coverage based on accessibility standards and learn what the best tool for which task is. Get insights and strategies for incorporating these tools into your web development projects!<\/p>\n\n\n\n

\"\"\/<\/figure>\n\n\n\n

Accessibility is an increasingly vital component of web development. <\/strong>It helps to minimize technological barriers and make websites and applications user-friendly and inclusive for all.<\/strong><\/p>\n\n\n\n

However, analyzing an entire website for accessibility issues can be a time-consuming and daunting task. That’s why we want to help you explore some tools. We will delve into two of the most popular options that can significantly assist you in enhancing your website’s accessibility.<\/strong><\/p>\n\n\n\n

We’ll put Pa11y-ci<\/strong><\/a> and axe <\/strong><\/a>+ <\/strong>WDIO<\/strong><\/a> under the microscope to evaluate their effectiveness in conducting automated accessibility testing. <\/p>\n\n\n\n

Our goal is to assess the capabilities and coverage of each tool concerning accessibility standards to deepen the accessibility testing. How will we achieve this? By examining a previous Proof of Concept (PoC) with axe + WDIO and undertaking a new PoC with Pa11y-ci.

We’ll also offer insights into their overall performance and suggest strategies for incorporating these tools into web development projects.<\/p>\n\n\n\n

<\/span>Overview – Pa11y-ci vs. axe + WDIO<\/strong><\/span><\/h2>\n\n\n\n

Pa11y-ci and axe + WDIO are both tools designed to help you validate web accessibility<\/strong>. They do this <\/strong>by utilizing a set of rules based on the Web Content Accessibility Guidelines (WCAG). <\/p>\n\n\n\n

These tools analyze the HTML code of your website and provide you with a detailed report of the findings. <\/strong>This can have various formats such as JSON or HTML. As a result, the team is able to identify and fix accessibility issues. <\/p>\n\n\n\n

To better understand what each tool brings to the table, we’ll go through a brief overview<\/strong> of both Pa11y-ci and axe + WDIO.<\/p>\n\n\n\n

<\/span>Pa11y-ci<\/strong><\/span><\/h3>\n\n\n\n

Pa11y-ci is an open-source command-line interface that makes it easy to run accessibility tests on your websites. <\/p>\n\n\n\n

Some of Its Key Features Include:<\/strong><\/p>\n\n\n\n

\u2705Automated Accessibility Testing.
\u2705Supports WAI-ARIA, WCAG 2.0, and Section 508 standards.
\u2705Easy integration with CI\/CD pipelines.
\u2705Ability to test multiple URLs concurrently.<\/p>\n\n\n\n

Below, you’ll find a visual representation of the Pa11y report. This image shows how the report provides a straightforward representation of the conducted accessibility tests. Please note that the report does not offer a comprehensive overview or detailed grouping of the identified issues. Instead, it serves as a convenient way to quickly assess the accessibility status of a site, highlighting the errors that were found.<\/strong><\/p>\n\n\n\n

\"\"\/
pally-ci html report<\/em> <\/figcaption><\/figure>\n\n\n\n

<\/span>axe + WDIO<\/strong><\/span><\/h3>\n\n\n\n

axe is an open-source library for automated testing of web page accessibility. Meanwhile, WebdriverIO (WDIO) is an automation framework for Node.js that provides capabilities for WebDriver protocol implementations like Selenium.
<\/strong>
Together, they offer a glimpse into a more powerful automation experience that takes accessibility into account.<\/strong> But here’s the thing: automation alone won’t cut it. We can’t just rely on it and brush aside the importance of manual testing.

The setup process of the framework was not primarily complex due to the integration with axe. This is as simple as importing the “@axe-core\/webdriverio” library. Instead, the challenge lies in installing a separate reporting tool to export the results to an HTML report.

Moreover, when utilizing a sitemap<\/strong><\/a> <\/strong>similar to Pa11y’s approach, it was necessary to employ Axios to fetch the URLs within the sitemap. Subsequently, a function had to be created to iterate the tests for each page found in the sitemap.<\/p>\n\n\n\n

The Perks of Using axe + WDIO Include:<\/strong><\/p>\n\n\n\n

\u2705Flexible Integration with WebdriverIO Test Framework.
\u2705Supports WAI-ARIA, WCAG 2.0, and Section 508 standards.
\u2705Detailed Reporting and Highlighting of Issues.
\u2705Ability to Test Dynamic and Complex Web Applications.

In the following, you’ll find several images of the axe + WDIO report. <\/strong>These images provide an overview of the results, followed by a list of accessibility issues found in the web application.

Additionally, the report includes sections that showcase results that pass the accessibility guidelines checks. And also results with incomplete and inapplicable accessibility results.<\/p>\n\n\n\n

\"\"\/
axe html report (Results Overview) <\/figcaption><\/figure>\n\n\n\n
\"\"\/
axe html report (Failed Details)<\/em> <\/figcaption><\/figure>\n\n\n\n
\"\"\/
axe html report  (Detailed Results Analysis)<\/em> <\/figcaption><\/figure>\n\n\n\n

<\/span>Automated Accessibility Testing: Comparison <\/strong><\/span><\/h2>\n\n\n\n

Below, we analyze the two tools based on the outcomes of conducting Proof of Concepts (PoCs).<\/p>\n\n\n\n

<\/span>axe + WDIO PoC<\/strong><\/span><\/h3>\n\n\n\n

\u2705On average a 57% coverage of WCAG accessibility issues<\/strong><\/a>.
\u2705Clearer reports, grouped issues, and linked solutions enhance result interpretation.
\u2705Enabled seamless integration with the WebDriver framework.
\u2705Provided a flexible test environment
\u274cMore complex configuration setup.<\/p>\n\n\n\n

<\/span>Pa11y-ci PoC<\/strong><\/span><\/h3>\n\n\n\n

\u2705Simplified execution through the command-line interface.
\u2705Allowed testing multiple URLs quickly. 
\u274cLimited capabilities compared to axe + WDIO.
\u274cLess precise in detecting all issues.<\/p>\n\n\n\n

<\/span>In a Nutshell – What is the Best Tool for Which Task?<\/strong><\/span><\/h2>\n\n\n\n

To sum up, both Pa11y-ci and axe + WDIO provide significant benefits for web automated accessibility testing. While Pa11y-ci stands out for its simplicity and speed, axe + WDIO offers more extensive capabilities.<\/p>\n\n\n\n

To support manual testing efforts, we recommend utilizing axe + WDIO <\/strong>as the primary method due to its comprehensive, categorized reports. This enables the prioritization of issues and the identification of areas meeting standards or requiring further review.<\/p>\n\n\n\n

For CI\/CD integration, Pa11y-ci is the superior choice<\/strong> as it is easy to set up and operates swiftly. It can be seamlessly integrated into the Build and Deploy stages to maintain a continuous emphasis on accessibility.

Are you interested in learning more about the suggested strategy for automated accessibility testing and implementing axe + WDIO?  Check out our proof of concept on <\/strong>GitHub<\/strong><\/a>. If you have any questions or need assistance, don’t hesitate to reach out!<\/p>\n\n\n\n

Looking for support in integrating Pa11y-ci into your CI\/CD pipeline?
Contact us<\/strong><\/a>! Let’s discuss how we can help you maintain a continuous focus on accessibility throughout your project’s development process.

Follow us on <\/strong>Linkedin<\/strong><\/a> & <\/strong>Twitter<\/strong><\/a> to be part of our community!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

Let’s dive into accessibility testing! We will explore efficient and effective tools that enhance website accessibility. Our goal is to evaluate their capabilities and coverage based on accessibility standards and learn what the best tool for which task is. Get insights and strategies for incorporating…<\/p>\n","protected":false},"author":76,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[354,60],"tags":[626,627],"yoast_head":"\nAutomated Accessibility: Comparing axe + WDIO and Pa11y-ci | Abstracta<\/title>\n<meta name=\"description\" content=\"Accessibility is an increasingly vital component of web development. It helps to minimize technological barriers and make websites and applications user-friendly and inclusive for all.\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Automated Accessibility: Comparing axe + WDIO and Pa11y-ci | Abstracta\" \/>\n<meta property=\"og:description\" content=\"Accessibility is an increasingly vital component of web development. It helps to minimize technological barriers and make websites and applications user-friendly and inclusive for all.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2023-07-19T16:33:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:19:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2023\/07\/Automated-Accessibility-Testing-Comparing-axe-WDIO-and-Pa11y-ci.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1250\" \/>\n\t<meta property=\"og:image:height\" content=\"705\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AbstractaUS\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/lh5.googleusercontent.com\/H4tyjYsvsvB3Kr8-NU0XJ8RJ42p2CKpX6oHY95hwOud5N4LW08G2NVosPuYVEOR7Bp2-t-OhlTfkkl4Ia1ANLjDSHObNo05mxRPrueVbaJokAtyrz_vsuKnkDmsGDYk2TEWf_MNaPbTeiSp_RSmFdF4\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\",\"name\":\"Automated Accessibility: Comparing axe + WDIO and Pa11y-ci | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/#primaryimage\"},\"datePublished\":\"2023-07-19T16:33:10+00:00\",\"dateModified\":\"2025-05-05T21:19:21+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/57d4a44a4680c705dcd51387e7977fea\"},\"description\":\"Accessibility is an increasingly vital component of web development. It helps to minimize technological barriers and make websites and applications user-friendly and inclusive for all.\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/\",\"name\":\"Accessibility Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/automated-accessibility-testing-comparing-axe-wdio-and-pa11y-ci\/\",\"name\":\"Automated Accessibility Testing: Comparing axe + WDIO and Pa11y-ci\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/57d4a44a4680c705dcd51387e7977fea\",\"name\":\"Leonardo Guedes\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/35a0f7458f7806b0cdb3aa0fdaf643e8?s=96&d=blank&r=g\",\"caption\":\"Leonardo Guedes\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15870"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/76"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=15870"}],"version-history":[{"count":4,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15870\/revisions"}],"predecessor-version":[{"id":15889,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15870\/revisions\/15889"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=15870"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=15870"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=15870"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}