Accessibility testing for beginners
Introduction
Accessibility is about everyone. Accessibility is about inclusion — making our products and services available, accessible and usable to all — serving every food moment for everyone.
It is also the way to bring attention to our Product by providing a better User Experience, better technical solutions and a great User Interface.
It is not an easy task and requires involvement from different teams: Designers, UX Researchers, Copywriters, Developers, Quality Assurance Engineers, Business Analysts, Product Owners, Management, etc.
When we build a new feature or change an existing one we should always keep in mind the importance of Accessibility.
This article will help you to understand how to start with Accessibility testing if you have never done it before.
Attitude
Many of us struggle when we are discovering something new, we experience resistance, learning is difficult, it requires time and patience to learn and adapt.
This struggle can happen also when first trying to test an app, a website, hardware, or anything else for accessibility compliance.
The most common question I hear is: I do not have any impairment so I can’t test it properly, this is likely the case for those of you who are just starting to learn about accessibility testing.
To ensure full testing beyond your capabilities, there is the option to involve independent and certified auditors. Though the help of such professional services is helpful it is not always possible, but there is already a lot of work you can do on your own and this is only a matter of time and effort, and the patience and empathy to learn how to test for accessibility.
Accessibility testing
Accessibility is a part of non-functional testing, which means accessibility testing answers the question “How does the system operate” rather than “What does our system do”.
Both manual and automated testing are possible.
Some tasks are difficult or impossible to automate, that is why manual work is required:
- During accessibility testing you can manually check that, screen readers actually work and provide the necessary information and that users are able to perform certain actions.
- You can provide feedback on user experience issues and usability.
- Keyboard navigation.
Let’s break down the whole testing process into stages:
Test planning and control
During this stage identify the scope and time budget for testing, ascertain which areas have the biggest impact on users, prepare a plan and estimate the effort. At this stage, we can also determine the Exit criteria.
During each further stage, you will also perform control and corrective activities to make your test efforts efficient.
Analyzing and Designing
Now when you have a plan and you identify the high-risk areas in your product that affect users the most, you can start analyzing what you will test and how. Start from requirements, try to apply them for all users.
As an example: For an app that sells tickets, the basic requirement for this app will be “to be able to find tickets by using the search input on the homepage”. Now you can check if it is easy to use the search input when you are a screen reader user, is the function of the input clear, does it explain itself, is it possible to search using only the keyboard, etc.
Many issues can be discovered at this step as you will see that requirements do not take into account all users.
After analyzing you can start designing your test cases and test approaches.
In this article, I am not going to go through all the test techniques but I want to emphasize that various techniques can be applied when you are trying to test accessibility, such as black box, static analysis or experience-based techniques.
You can combine them together, do not hesitate to experiment and let this testing be your art project.
Implementation and Executing
During this stage, we create test cases and execute them in a sequence which we planned or which suits our project and needs the most. My personal advice: make use of existing accessibility assessment tools during your test execution phase. Please note that accessibility assessment tools can’t completely replace manual testing. Accessibility assessment tools will give you knowledge about the issues related to color contrast, missing labels, attributes, roles, missing translations or even useless or absent button/links names. They will provide you with a description of the problem, link to the violated guideline and even offer you a solution on how to fix the problem, but for keyboard navigation or screen readers flow you will have to use manual testing.
With this information, you can start creating reports and start discussions within the team regarding priorities.
One of the most popular tools like Lighthouse or WAVE is open source and free to integrate and use.
The list of the tools can be extended and it depends on your experience with the tool and your objectives.
Evaluating Exit criteria and reporting
At this stage we need to understand for a certain test level if we performed enough testing and can we stop or should we do more testing. It is important to understand that Exit criteria should be based on a risk assessment which was performed in the earlier stages and on the time which you have according to the project deadlines.
You can use conformity with WCAG2.1 guidelines as Exit criteria as well. For example, ask yourself a question if your test object has an appropriate accessibility level according to WCAG2.1
The next step is to create a report to stakeholders and the team about the level of accessibility which your product has.
Test Closure activities
When the testing has been completed we need to archive testware, gather metrics and check that we created sufficient summary reports for stakeholders. We also should finalize all test scripts and plans. If you are going to transfer your testware further to a third-party organisation or another team then make sure that everything is accurate and has a prepared description.
As you can probably notice the stages I described are not new because they are repeating the test process described in the ISTQB Glossary.
Tools for accessibility testing
There are many tools which can help us in our accessibility testing journey.
Some tools provide a very detailed report and in case you are unsure about what to test and how these tools can be a good starting point on what to check.
I am not going to go through all available tools (probably because I do not know them all) but I will give you a couple of examples that can be very useful.
WAVE Web Accessibility Evaluation Tool
WAVE is a suite of evaluation tools that helps authors make their web content more accessible to individuals with disabilities. WAVE can identify many accessibilities and Web Content Accessibility Guideline (WCAG) errors, but also facilitates human evaluation of web content.[1]
This tool also offers an extension in Chrome and Firefox. The use is very simple, you just add an extension to your browser, visit the web page which you want to evaluate and start evaluation by clicking on the extension.
If you prefer to use the online web version, you can visit the WAVE WebAim website and enter the url of your web page there. Please be aware that if your web page is hidden behind a firewall or any other protection from open web traffic (e.g Cloudflare, Amazon VPC) then probably the online webpage of WAVE won’t be sufficient for you and you have to use the extension instead.
The report from WAVE will have a lot of information including links to the place where the error occurred, the name of the error, violated guidelines and an explanation why the following error appeared, what it means and how it can be fixed.
This report done with WAVE is already very extensive. This will allow you to create a detailed report on specific accessibility issues found. This can also be added to integration tests as WAVE provides an API, yet the API calls are not free.
Lighthouse
Lighthouse is an open-source, automated tool for improving the quality of web pages. You can run it against any web page, public or requiring authentication. It has audits for performance, accessibility, progressive web apps, SEO and more.[2]
Lighthouse now is built-in to the Developers tools in the Chrome browser. You can find it by opening Developer tools > Lighthouse.
Tests run on actual URLs — meaning that virtual navigation and SPAs are harder to test if they can’t be accessed through a dedicated URL.
This tool provides a means to check your webpage for both Mobile and Desktop views. Lighthouse can evaluate the accessibility of a web page and calculate a score based on valid and violated accessibility issues and WCAG rules. This given score is relative based on the checked guidelines. Many accessibility issues Lighthouse can not assess though. Therefore it’s important to keep in mind that a high score does not automatically mean that the page is actually truly accessible.
The Lighthouse accessibility score is a weighted average of all accessibility audits. Weighting is based on axe user impact assessments.[3] Lighthouse uses a set of rules which are based on axe application. So the score is based on the amount that is evaluated successfully. If the rule does not meet then the average score will be lower.
Lighthouse can provide detailed information on violations which were made and recommend the best options to fix the issue. However, Lighthouse won’t give a reference to the violated guideline which requires additional investigation from your side, but the information should be enough to create issue reports and begin accessibility improvement on the page.
Lighthouse also provides the solution with CI integration which is free.
Axe DevTools
Axe DevTools is an extension that can be added to Chrome as a separate tab in developer tools.
This tool will evaluate your page or a specific element in your page (this option is also available) and provide a report about found issues, the severity of the issue and recommendations on how to fix it. If you want to find the exact guideline point violated, you will need to spend a bit more time investigating because axe does not provide a clear understanding of the point violated. But the information provided is usually enough to create your own issue report or to find violated guidelines by yourself.
Axe is also well known and valued because of its wide range of options to be integrated with your CI and make accessibility check as an automated process.
Please note, the same as with Lighthouse, the error absence in the axe does not mean that your product is accessible.
The tools described above should be enough to start accessibility evaluation in your project.
Nevertheless, I encourage you to learn more about these tools and maybe find more information about other tools which can be efficient in your particular project and, as I mentioned in the beginning, manual accessibility testing should be performed together with these tools.
Screen readers
This part of my article I want to dedicate to Screen readers. These tools are very important for accessibility testing and without checking your product with screen readers, you can’t ensure that it is truly accessible and inclusively usable.
Why is that?
Screen readers are tools used by people with visual impairments. Visual impairment is one of the most common disability types in the world. According to the World Health Organisation globally, at least 2.2 billion people have near or distant vision impairment. In at least 1 billion — or almost half — of these cases, vision impairment could have been prevented or has yet to be addressed.[4]
For people with any kind of visual impairment, screen readers are very helpful.
The most popular screen readers are: NVDA, JAWS, VoiceOver, ZoomText/Fusion, System Access or SA To Go, Narrator, ChromeVox, etc.
The tricky part is that different screen readers can behave differently amongst themselves and for different devices.
It does not mean you have to test each screen reader, yet you should rather determine which screen reader selection could make sense to test for your project.
The top three: NVDA, JAWS and VoiceOver, but worth mentioning that there is no official tracked usage data on this, but only voluntarily provided user input on surveys.
My tips for screen readers testing are:
Tip 1: Learn your screen reader
Do not try to start testing right away. If you are not used to a screen reader you will face some obstacles in using it. It will feel weird, uncomfortable, maybe even irritating. Give yourself time.
Learn some shortcuts keys or touch gestures which are used for a specific screen reader, watch videos on how people with visual impairments use screen readers.
Step-by-step you will feel more comfortable and then you can start accessibility testing on your project with screen readers.
Tip 2: Check different options of screen reader’s interactions with elements.
Screen readers can have a couple ways of reading content on the page — for example using keyboard navigation or mouse/finger hover.
Both ways should be checked during accessibility testing.
Tip 3: Check both mobile and desktop environments.
Sometimes it happens that the flow or order of elements on the desktop is different from the one on mobile. As a result, the screen reader is able to read elements in one environment but completely fails in another. It can also happen because different screen readers are being used for mobile and desktop. Even VoiceOver iOS and VoiceOver macOS have noticeable differences sometimes.
Tip 4: Close your eyes
Sounds weird but sometimes it can help. You can use “curtains” on your device from Apple (which turns off the screen and you can orient only with a screen reader) or “Dark screen” for some Android devices. For Windows users there is no such function as a screen curtain.
When you do not see the image on your screen then you will be oriented only on the sound and text which the screen reader will read to you. Additionally, you will have the same experience as a visually impaired person when you get stuck in the keyboard trap or when some functions are not responding to your actions.
Tip 5: Understand your objectives
Testing with or without screen readers requires understanding what you are going to test. Before you start, just analyze what you want to achieve? What does this specific feature do? What is the expected outcome? When you have answers to these questions it is easy to apply them to the screen readers workflow. For example, you test a button which should add something to your basket. The same behavior will be expected when you are a screen reader user.
Another example is validation errors. When you are a user with no visual impairments then it goes without saying that you can see clearly when something went wrong and you can perform some actions to fix that. Will it be the same for screen reader users? Will they understand that something went wrong?
Conclusions
Accessibility testing is a part of your Quality processes in your work. It requires some specific knowledge and attitude. But as with any other testing activity: You can learn and train yourself to perform accessibility testing.
As a part of non-functional testing, accessibility testing also has different steps and can make use of different testing techniques.
You can rely on WCAG2.1 as a guideline for your testing and the expected outcome.
Accessibility assessment tools can provide you a lot of benefits and they are a good start to perform accessibility evaluation in your project. Most of the tools are free and can be included in automated testing.
One of the important tools for accessibility testing are screen readers.
Unfortunately, for now, there is no way to automate screen readers' behavior, which is why manual testing should be performed.
Screen readers require some time to adapt to them and require some training to work with them. All these are achievable skills and you do not have to be a screen reader user.
Including people with different impairments in the testing process and development process is very beneficial and can help you to avoid a lot of issues at the very beginning.
Resources
- https://wave.webaim.org
- https://developers.google.com/web/tools/lighthouse
- https://web.dev/accessibility-scoring/
- https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment
—
Just Eat Takeaway.com is hiring. Apply today!
