Automated web accessibility testing with Nightwatch.js and axe

January 20, 2020

#post

36

#software testing

18

#quality assurance

17

#nightwatchjs

15

#accessibility testing

2

Web Content Accesibility Guidelines and the ADA

Testing for compliance with web content accessibility guidelines or WCAG is becoming increasingly important. Legal precedent on how the Americans with Disabilities Act applies to websites and apps is finally catching up with the web.

In the infamous Dominos Pizza case the 9th circuit court stated,

We agree with the district court in this case—and the many other district courts that have confronted this issue in similar contexts—that the ADA applies to Domino's website and app, which connect customers to the goods and services of Domino's physical restaurants.

I'm strongly of the opinion that web content accessibility guidelines are good design principles and are good to follow regardless of the threat of legal implications.

So what accessibility testing tools can we use to bake accessibility testing into our selenium test suites?

Nightwatch.js and Axe for Accessibility Testing

Nightwatch is a great browser automation framework. Axe is an accessibility rule verification tool (it is even used Chrome Lighthouse).

Wouldn't it be great if the two could be combined? Some people already thought of that so I built upon their work to do automated web accessibility testing in my existing automation suite.

I found a couple npm packages that let you inject axe-core into Nightwatch as a custom command, but one didn't work with newer Chrome versions and the reporting on the other wasn't verbose enough for me.

I built on that pattern with nightwatch-axe-verbose npm package.

npm install nightwatch-axe-verbose -s

Writing Automated Accessibility Tests

To write Nightwatch tests to perform accessibility assertions with nightwatch-axe-verbose you need to add to your nightwatch.json file the location of the package inside the custom commands area

"custom_commands_path": ["./node_modules/nightwatch-axe-verbose/src/commands"]

This will allow you to use axe assertions in your test.

With the accessibility testing package added as a custom command you can use it in your Nightwatch.js tests. Here are some example accessibility tests below.

module.exports = {
    '@tags': ['accessibility'],
    'Inaccessible site': function (browser) {

        browser
            .url('https://www.w3.org/WAI/demos/bad/before/home.html')
            .assert.title('Welcome to CityLights! [Inaccessible Home Page]')
            .axeInject()
            .axeRun('body', {
                'color-contrast': { enabled: false },
            })
            .end();
    },
    'Accessible site': function (browser) {

        browser
            .url('https://www.w3.org/WAI/demos/bad/after/home.html')
            .assert.title('Welcome to CityLights! [Accessible Home Page]')
            .axeInject()
            .axeRun('body', {
                rules: {}
            })
            .end();
    }
}

.axeInject() makes the custom commands available and .axeRun() will execute all the tests in the axe-core ruleset unless customized. In the examples above, the first test provides an example of how to disable specific rules or you can specify specific rules *to run as well.

Useful test case output

It is called -verbose because the package will report each passing accessibility rule run as a test case along with a count of how many controls the rule was run against. This is to ensure confidence in the results.

Passing test output

√ Passed [ok]: aXe rule: aria-allowed-role (2 elements checked)
√ Passed [ok]: aXe rule: aria-hidden-body (1 elements checked)
√ Passed [ok]: aXe rule: aria-hidden-focus (156 elements checked)
√ Passed [ok]: aXe rule: aria-required-attr (2 elements checked)
√ Passed [ok]: aXe rule: aria-required-children (2 elements checked)
...

Each failing test is reported per control per rule to ensure you get a complete picture of what is violating WCAG accessibility on your site so you can fix it all at once. Other libraries I was looking would exit on first failure, hiding downstream issues.

Failing test output

× Failed [fail]: (aXe rule: heading-order - Heading levels should only increase by one  
        In element: h5
× Failed [fail]: (aXe rule: image-alt - Images must have alternate text
        In element: .small-logo
× Failed [fail]: (aXe rule: label - Form elements must have labels
        In element: #SearchInput)

Nightwatch accessible test project on my GitHub