Accessibility Testing — An underrated topic

8 min readMar 1, 2024

Are you aware that nearly 1.3 billion people worldwide experience significant disability?

People with disabilities should have the same right to have a good experience when interacting with the digital world. It is a matter of equal rights to ensure accessibility.

What if I tell you that by 2030, almost 17% of the world’s population will be over 65 years old?

This topic is something that should be addressed, and sometimes, the only goal is to deliver something that works, and that meets users/business requirements.

But don’t forget that any company could develop the same product just like yours, and sometimes the trust of a customer is won with non-functional aspects like accessibility and performance.

Also, companies that address this topic are free from being involved in lawsuits….

Last year (2023), 77% of lawsuits were filed against organizations with under $25 million in revenue.

The penalty for violating these regulations ranges from $75,000 to $150,000, a significant amount for burgeoning companies, particularly when contrasted with the expense of ensuring website accessibility. 😟

In order to ensure website accessibility, there are some regulations that you can follow, such as the Web Content Accessibility Guidelines (WCAG), ADA (American Disabilities Act), and Section 508, which is a United States law that requires federal agencies and some federal contractors to make their websites accessible.

In this article, I will address WCAG in more detail, but feel free to explore the guidelines from other regulations, as they might be useful depending in your location.

And also, I will provide you a tool to help you meet those standards and make your product more compliant with accessibility concerns. 😃

Table of Contents

What is WCAG?
Which is the WCAG success criteria?
How can I make my product WCAG compliant?
How can I write theses tests with Playwright?
How can I scan the page for specific violations?
How can I handle known accessibility issues?
How about its test execution report?
Conclusion

What is WCAG?

These guidelines are a set of standards designed to enhance web content accessibility for individuals with disabilities, and are particularly crucial for businesses serving the public as they ensure inclusivity for all users.

Published by the Web Accessibility Institute (WAI), WCAG provides designers, developers, and others involved in digital product creation with a user-friendly framework.

This framework helps ensure that products adhere to recognized standards and best practices for accessibility. By striving for specific levels of WCAG compliance, such as level AA or higher (discussed shortly), designers can fulfill Section 508 standards for their websites.

WCAG encompasses a range of requirements, ensuring that websites are:

Perceivable: Content must be easily perceivable by individuals with various disabilities. This involves considerations such as contrast and color, providing descriptive text for non-text content like images, videos, and animations, and offering alternatives for voice-over text.
Operable: Websites must be operable through keyboard navigation and compatible with assistive technologies like screen readers and point sticks, allowing users to interact effectively.
Understandable: Content organization and labeling must adhere to specific criteria, ensuring clarity and comprehension. Text should be understandable for speakers of the language presented and also for those who may require translation, such as into English.
Robust: The content should be robust to ensure reliable interpretation by diverse user agents, including assistive technologies. This entails utilizing well-supported technologies, steering clear of deprecated or obsolete features, and ensuring compatibility with both current and upcoming technologies.

Which is the WCAG success criteria?

The WCAG Success Criteria are categorized into three levels of compliance: A, AA, and AAA.

A: Minimal Compliance This level provides basic accessibility, though it may not meet all web and regulatory standards. Key A compliance guidelines include:

Absence of keyboard traps
Navigability with a keyboard
Provision of non-text content alternatives
Implementation of video captions
Avoidance of conveying meaning solely through shape, size, or color

AA: Acceptable Compliance AA compliance is the standard for ensuring a usable experience for all users and meeting regulatory standards like Section 508. Essential requirements at this level include:

Color contrast of at least 4.5:1 in most cases
Usage of alt text or similar for meaningful images
Consistent navigation elements across the site
Accurate labeling of form fields
Screen-reader-compatible status updates
Logical order of headings

AAA: Optimal Compliance Achieving AAA compliance often requires specialized expertise. Meeting these standards broadens audience reach and ensures public accessibility requirements are met. Some AAA standards include:

Sign language interpretation for audio or video content
Higher color contrast ratio (at least 7:1 in most cases)
Removal of timing as a crucial part of activities
Availability of context-sensitive help

How can I make my product WCAG compliant?

Meeting web accessibility standards can present challenges due to the varying criteria set for different facets of a website.

The ideal panorama is to have this verification process automated, and though the solution I’m about to present you is capable of finding common accessibility problems, it doesn’t eliminate the need of complementar manual testing.

Good news are that Playwright can be used to make this process automated!

Together with the usage of the @axe-core/playwright package, the test will be able to find common problems, such as the ones presented in the list below and many others:

Text with insufficient color contrast against its background, making it difficult for users with vision impairments to read.
User interface controls and form elements lacking labels, thereby hindering identification by screen readers.
Interactive elements featuring duplicate IDs, potentially causing confusion for assistive technologies.

How can I write theses tests with Playwright?

The good news are that accessibility tests work just like any other Playwright test!

I’ll give you a really simple example of its usage:

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright'; // 1

test.describe('homepage', () => { // 2
  test('should not have any automatically detectable accessibility issues', async ({ page }) => {
    await page.goto('https://your-site.com/'); // 3

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze(); // 4

    expect(accessibilityScanResults.violations).toEqual([]); // 5
  });
});

Following the step numbers mentioned in the commented lines:

Import the @axe-core/playwright package.
Use standard Playwright Test syntax to establish a test case.
Employ typical Playwright syntax to navigate to the page being tested.
Execute AxeBuilder.analyze() to conduct an accessibility scan on the page.
Apply regular Playwright Test assertions to confirm the absence of violations in the scan results.

And this is in the scenario that you want to scan the whole page.

If you want to only target specific elements of a page, use the AxeBuilder.include() method:

const accessibilityScanResults = await new AxeBuilder({ page })
      .include('#your-element-id')
      .analyze();

How can I scan the page for specific violations?

By default, the validations are done against a wide variety of accessibility rules. You can specify which standard or group of standards that you want to include in you analysis, as shown in the table below:

This can be achieved by simply using the AxeBuilder.withTags()method:

const accessibilityScanResults = await new AxeBuilder({ page })
      .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
      .analyze();

How can I handle known accessibility issues?

When incorporating accessibility tests into an application, a common concern arises: how to handle known violations effectively. Below are several techniques to address this issue:

Excluding Individual Elements from Scans

If your application contains specific elements with known issues, you can utilize AxeBuilder.exclude() to omit them from scans until they are rectified. However, note the following drawbacks:

Excluding elements and their descendants: Be cautious when using exclude() with components having numerous children.
Prevention of all rule execution: exclude() prevents all rules from running against the specified elements, not just those corresponding to known issues.

const accessibilityScanResults = await new AxeBuilder({ page })
    .exclude('#element-with-known-issue')
    .analyze();

Disabling Individual Scan Rules

If your application exhibits multiple violations of a specific rule, you can employ AxeBuilder.disableRules() to temporarily deactivate those rules until they are addressed. Rule IDs can be found in the id property of the violations.

const accessibilityScanResults = await new AxeBuilder({ page })
    .disableRules(['duplicate-id'])
    .analyze();

You can find a complete list of these axe’s rules can be found here.

Using Snapshots for Specific Known Issues

For a more detailed approach to known issues, snapshots can be utilized to verify that pre-existing violations remain unchanged. This method avoids the limitations of AxeBuilder.exclude() but requires slightly more complexity.

expect(violationFingerprints(accessibilityScanResults)).toMatchSnapshot();

// my-test-utils.js
function violationFingerprints(accessibilityScanResults) {
  const violationFingerprints = accessibilityScanResults.violations.map(violation => ({
    rule: violation.id,
    targets: violation.nodes.map(node => node.target),
  }));

  return JSON.stringify(violationFingerprints, null, 2);
}

By employing these techniques, you can effectively manage and address known accessibility issues within your application.

How about its test execution report?

By default, a not so user friendly error message is shown after the test execution:

But, do not worry! There is a more viable option, and only requires using the axe-html-reporter library to generate a useful HTML report.

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { createHtmlReport } from 'axe-html-reporter';
const fs = require('fs');

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test.describe('Playwright Homepage', () => {

  test('Verify there is no accessibility issues', async ({ page }) => {

    await page.goto('https://playwright.dev');

    const accessibilityScanResults = await new AxeBuilder({ page }).analyze();

    const reportHTML = createHtmlReport({
      results: accessibilityScanResults,
      options: {
        projectKey: "PlaywrightHomepage"
      },
    });

    if (!fs.existsSync("build/reports/accessibility-report.html")) {
      fs.mkdirSync("build/reports", {
        recursive: true,
      });
    }
    fs.writeFileSync("build/reports/accessibility-report.html", reportHTML);

    expect(accessibilityScanResults.violations).toEqual([]);


  });

});

Furthermore, the HTML report generates a table with the problems found:

And for each violation, you can check more details, and even how to address that problem:

Conclusion

This article has explored various aspects of web accessibility testing using the Axe accessibility testing engine in conjunction with Playwright.

By following the outlined techniques and examples, you can automate accessibility tests, identify common accessibility issues, and generate detailed reports for further analysis.

However, it’s important to recognize that automated testing is not a substitute for manual testing, and a combination of both approaches is recommended for comprehensive accessibility coverage.

By integrating these practices into your development workflow, you can ensure that your websites and applications adhere to accessibility standards, providing a more inclusive experience for users of all abilities.

I hope you enjoyed reading this article!

My name is João Coelho, and I am currently a QA Automation Engineer at Talkdesk. Lately, I have been writing articles regarding automation, QA and software engineering topics, that might not be known by the community.

If you want to follow my work, check my Linkedin and my author profile at Medium!

Furthermore, if you’re interested in further supporting me and my content creation efforts, you can do so by buying me a coffee! 😄👇

Your support goes a long way in helping me dedicate more time to researching and sharing valuable insights about automation, QA, and software engineering.