{"id":10055,"date":"2018-07-09T22:38:52","date_gmt":"2018-07-09T22:38:52","guid":{"rendered":"http:\/\/abstracta.us\/blog\/?p=10055"},"modified":"2025-05-05T21:23:44","modified_gmt":"2025-05-05T21:23:44","slug":"accessibility-testing-in-continuous-integration","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/","title":{"rendered":"How to Easily Do Accessibility Testing in Continuous Integration"},"content":{"rendered":"<h1><span style=\"font-weight: 400;\">How to add accessibility tests to your CI pipeline using the open source tool, Pa11y<\/span><\/h1>\n<p><span style=\"font-weight: 400;\">If you are unfamiliar with continuous integration (CI), it\u2019s a practice wherein <\/span><span style=\"font-weight: 400;\">each developer\u2019s code is merged frequently (at least once per day). In this way, a stable code repository is maintained from which anyone can start working on a change. The build is automated with various automatic checks, such as code quality reviews, execution of unit tests, etc. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Of the tests that can be run in a continuous integration environment, we will be looking at accessibility tests. Accessibility testing, according to <\/span><a href=\"https:\/\/www.guru99.com\/accessibility-testing.html\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">Guru99<\/span><\/a><span style=\"font-weight: 400;\">, is a subset of usability testing, and it\u2019s performed to ensure that the application being tested is usable by people with disabilities like the deaf, blind, and color blind as well as other disadvantaged groups.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To perform accessibility tests, you can use the open source tool, <a href=\"https:\/\/github.com\/pa11y\" target=\"_blank\" rel=\"noopener\">Pa11y<\/a> which lets you know if an application meets certain accessibility standards or not. At the same time, Pa11y allows a simple integration with CI engines like Circle-CI, Jenkins or Travis-CI, which is why it\u2019s a natural choice.<\/span><\/p>\n<p><a href=\"https:\/\/github.com\/pa11y\"><img decoding=\"async\" class=\"aligncenter wp-image-10056 size-full\" src=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/pa11y-logo.jpg\" alt=\"pa11y\" width=\"360\" height=\"200\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">There are different versions of the tool with which you can work: by command line, by a web service, and others. For this particular example, we\u2019ll use the console version.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">It\u2019s important to mention that for the proper functioning of Pa11y, you must have installed Node.js 4+ and PhantomJS. Then it is installed by command line very simply, with the following command:<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0npm install -g pa11y<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">To use pa11y, use this command:<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\"><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y [options] &lt;url&gt;<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">The options field allows you to customize the type of testing that is going to be carried out in many ways. Among the parameters that can be specified, the main ones are:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>-s, -standard &lt;name&gt;<\/b><span style=\"font-weight: 400;\">: The accessibility standard is selected with which the test will be carried out. By default, it uses WCAG2AA. Other options include: Section508, WCAG2A, WCAG2AAA.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>-r, -reporter &lt;reporter&gt;<\/b><span style=\"font-weight: 400;\">: Refers to how the results obtained are returned. If this option is not included, the results are displayed by console. The available format options include: cli, csv, tsv, html and json.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>&#8211; l, -level &lt;level&gt;<\/b><span style=\"font-weight: 400;\">: The message level with which it fails (error, warning, notice).<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>-i, -ignore &lt;ignore&gt;<\/b><span style=\"font-weight: 400;\">: Allows you to specify certain things that you want to ignore. For example, warning messages may be ignored, or a certain principle of the standard may be ignored for some reason.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>-T, -threshold &lt;number&gt;<\/b><span style=\"font-weight: 400;\">: Number of errors, warnings, or notices allowed without making the test fail.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">There are many more configurations that you can have, which can be found on the tool&#8217;s <\/span><a href=\"https:\/\/github.com\/pa11y\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">website<\/span><\/a><span style=\"font-weight: 400;\">. To do some basic testing (without any particular configuration, taking the ones that come by default), you can use this command:<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\"><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y<\/span> <span style=\"font-weight: 400;\">http:\/\/myexamplepage.com<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">If you want your testing to be more stringent, following the WCAG2AAA standard, use this command:<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\"><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y &#8211;standard WCAG2AAA<\/span> <span style=\"font-weight: 400;\">http:\/<\/span><span style=\"font-weight: 400;\">\/myexamplepage.com<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">If you also want the results to be saved in a file (csv) to later consult them:<\/span><\/p>\n<p><em><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y &#8211;reporter csv http:\/\/myexamplepage.com &gt; myresults.tsv<\/em><\/p>\n<p><span style=\"font-weight: 400;\">If you want to allow 10 errors without the test failing:<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\"><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y &#8211;threshold 10 &#8211;reporter csv<\/span> <span style=\"font-weight: 400;\">http:\/\/<\/span><span style=\"font-weight: 400;\">myexamplepage.com<\/span><span style=\"font-weight: 400;\"> &gt; misresultados.tsv<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">Regarding the exit codes, pa11y has three:<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">0: Pa11y was executed successfully, and there are no errors.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">1: Pa11y failed due to technical failure.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">2: Pa11y ran successfully but there are errors on the page.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">For example, I ran the command this way from my Travis-CI (configuring the command in the .travis.yml file):<\/span><\/p>\n<p><em><span style=\"font-weight: 400;\"><em><span style=\"font-weight: 400;\">\u00a0 \u00a0 \u00a0<\/span><\/em>pa11y &#8211;threshold 30<\/span> <span style=\"font-weight: 400;\">open-cart.azurewebsites.net<\/span><span style=\"font-weight: 400;\"> -s WCAG2AA<\/span><\/em><\/p>\n<p><span style=\"font-weight: 400;\">In this example, I set a threshold of 30 errors. Therefore, the test only fails if the page being tested has more than 30 errors. Although the test doesn\u2019t fail if it has fewer errors, you can see them all in detail in the execution of Travis-CI in the output of the build. The WCGAA standard was used.<\/span><span style=\"font-weight: 400;\"><br \/>\n<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Here\u2019s an example of the detail of the errors obtained:<\/span><\/p>\n<p><a href=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/accesibilidad.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-10057\" src=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/accesibilidad.png\" alt=\"pa11y error\" width=\"926\" height=\"99\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">This error indicates that the contrast of colors on the page is not adequate. However, by the threshold that I set, the test didn\u2019t fail.<\/span><\/p>\n<p><a href=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/pa11y.png\"><img decoding=\"async\" class=\"aligncenter size-full wp-image-10058\" src=\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/pa11y.png\" alt=\"pa11y results\" width=\"601\" height=\"142\" \/><\/a><\/p>\n<p><span style=\"font-weight: 400;\">The test ends with exit code 0, which indicates that it was executed successfully and there were no errors (above the threshold I set). <\/span><\/p>\n<p><span style=\"font-weight: 400;\">For more options, check these guides: <\/span><a href=\"https:\/\/github.com\/pa11y\/pa11y-ci\" target=\"_blank\" rel=\"noopener\"><span style=\"font-weight: 400;\">https:\/\/github.com\/pa11y\/pa11y-ci<\/span><\/a><\/p>\n<p><span style=\"font-weight: 400;\">And there you have it, this is how you can easily add accessibility tests into your CI pipeline, considering the special needs your users may have to make your product accessible to a wider and more diverse user base! <\/span><\/p>\n<p><span style=\"font-weight: 400;\">Have you tried this or something more advanced? Let me know how it went! <\/span><\/p>\n<hr \/>\n<h2><strong>Recommended Posts<\/strong><\/h2>\n<p><a href=\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-accessibility-testing\/\"><span style=\"font-weight: 400;\">[Infographic] What is Mobile Accessibility and Why is it Important?<\/span><\/a><br \/>\n<a href=\"https:\/\/abstracta.us\/blog\/mobile-testing\/use-monkop-continuous-mobile-testing\/\"><span style=\"font-weight: 400;\">How to Use Monkop for Continuous Mobile Testing<\/span><\/a><\/p>\n<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>How to add accessibility tests to your CI pipeline using the open source tool, Pa11y If you are unfamiliar with continuous integration (CI), it\u2019s a practice wherein each developer\u2019s code is merged frequently (at least once per day). In this way, a stable code repository&#8230;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[354],"tags":[118,223],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Accessibility Testing in Continuous Integration | Abstracta<\/title>\n<meta name=\"description\" content=\"Using Pa11y, you can achieve accessibility testing in continuous integration in a pinch, making your app all the more user friendly. Read how!\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Accessibility Testing in Continuous Integration | Abstracta\" \/>\n<meta property=\"og:description\" content=\"Using Pa11y, you can achieve accessibility testing in continuous integration in a pinch, making your app all the more user friendly. Read how!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-09T22:38:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:23:44+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/accessibility-testing-ci.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"560\" \/>\n\t<meta property=\"og:image:height\" content=\"315\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@fltoledo\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/abstracta.us\/wp-content\/uploads\/2018\/07\/pa11y-logo.jpg\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\",\"name\":\"Accessibility Testing in Continuous Integration | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/#primaryimage\"},\"datePublished\":\"2018-07-09T22:38:52+00:00\",\"dateModified\":\"2025-05-05T21:23:44+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\"},\"description\":\"Using Pa11y, you can achieve accessibility testing in continuous integration in a pinch, making your app all the more user friendly. Read how!\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/\",\"name\":\"Accessibility Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\",\"url\":\"https:\/\/abstracta.us\/blog\/accessibility-testing\/accessibility-testing-in-continuous-integration\/\",\"name\":\"How to Easily Do Accessibility Testing in Continuous Integration\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\",\"name\":\"Federico Toledo, Chief Quality Officer at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6de7ec6536c4028b5c02ad4ec1b9af0d?s=96&d=blank&r=g\",\"caption\":\"Federico Toledo, Chief Quality Officer at Abstracta\"},\"description\":\"Co-founder and COO of Abstracta\",\"sameAs\":[\"https:\/\/twitter.com\/fltoledo\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/10055"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=10055"}],"version-history":[{"count":9,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/10055\/revisions"}],"predecessor-version":[{"id":16027,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/10055\/revisions\/16027"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=10055"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=10055"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=10055"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}