An evaluation of automation: benefits and dangers of accessibility tools

Automated evaluation tools can be useful in providing a general accessibility overview as long as the dangers are kept in mind.

Here's the scene: it's 4:45pm and after presenting for a full day at a web accessibility workshop, the end is in sight. Everything has gone so well — the audience were engaged in the screen reader demonstration, there was great enthusiasm about making their own websites WCAG 2.0 compliant, and best of all, the audience seemed to genuinely understand that they can make a real difference to the lives of people with disabilities. As the day draws to a close, you provide one last opportunity for questions.

And then it comes.

"What's the easiest way to check if my website is accessible?"

You knew it was coming, but you suddenly feel a bit queasy and turn pale. One half of you worries about how to answer such a complicated question in a few minutes, and the other half feels dejected because the question shouldn't be complicated. You start to explain about the tools that are available, but then it devolves into a rushed explanation of what works, what doesn't and what else needs to be done. The eyes of the person who asked the question start to glaze over, while the others in the room stare at the clock, groaning as they realise how loaded the question turned out to be.

While most accessibility specialists can relate to this scenario, it's important to acknowledge that the question is a really good one, and important one. It is always a hotly discussed topic in the Professional Certificate in Web Accessibility Compliance that I co-teach, and it's my hope that this month's column can provide some practical guidance around when the tools are useful but also some of the dangers associated with the automation process.

Firstly, it's worth defining what we're talking about in relation to automated tools. Also known as evaluation tools or validation tools, they are software programs or websites that allow you to run a computerised assessment on the code of a webpage to check its accessibility. The tools range from testing one webpage through to every webpage on a website, and the costs range from free to very commercial software prices for enterprise-based solutions. Access iQ™ has a great resource on web accessibility tools should you want to try one out. Traditionally, the tool will ask you for a website address and then provide you with an accessibility report outlining the errors it picked up based on the WCAG guidelines.

Benefits of accessibility tools

So what are the main benefits of using an automation tool? Here's my top five:

1. It's a great heads-up as to the accessibility of your website. The tools evaluate all the things that a computer can check including things like the presence of alternative text, comparison of colour values, labels in tables, and coding errors just to name a few. If you run a tool on your website and it comes back with pages and pages of errors, it's reasonable to presume that the website will need some accessibility work. If it only shows a few, it's likely that many parts of the website are accessible. Many people turn to evaluation tools to get a feel as to whether a website can be retrofitted to be accessible or whether it's easier to start from scratch.
2. Helpful in identifying some of the big issues. Many high-priority issues such as alternative text are picked up in an evaluation tool and depending on how good the tool is, it can break down code and provide very specific information on what the problem is and how to fix it, saving a lot of time in the process.
3. Allows you to check the website against many different accessibility levels and standards. Most tools will give you a choice as to whether you want to check your website against WCAG 2.0 Level A, AA, AAA; Section 508, and a variety of other web standards. This can be very helpful in specifically targeting the evaluation to the particular requirement in the environment that you're working in. It can also be helpful in checking issues such as web browser compatibility, which would take a long time to do manually.
4. It's quick and easy for ICT decision-makers. Information and communications technologies (ICT) decision-makers often comment that a good tool is useful in that within a few minutes, or a few hours for a whole website, you can assess a website and then pass the results onto ICT professionals for implementation.
5. The tools evolve as the standards and technologies evolve. With work continuing around the Evaluation and Report Language (EARL) 1.0 scheme in the W3C Evaluation and Repair Working Group and the evaluation methodologies being created by the Research and Development Working Group, the effectiveness of evaluation tools is only going to get better in time.

Dangers of accessibility tools

While using an evaluation tool is useful, it's extremely important that people understand that such tools are only the beginning of the story, not the end of the story. Here are some issues that need to be considered when using an evaluation tool:

1. Tools do not check everything. Given that tools are limited by how a computer processes code, there are many accessibility elements that they simply can't verify. Do videos have audio description? Are there images on the website that could cause a seizure? Does the website work in predictable ways? These are just a few of the issues that a tool will not currently pick up and therefore additional evaluation is required.
2. Different tools produce different results. As with all computer software, the tool will work differently depending on how the software developers approached the task. There are also some elements on websites, such as JavaScript, that can have strange effects on evaluation tools. As such, any report created by a tool should be taken as a guide only, not as an infallible representation of website accessibility.
3. Tools can be a bit dramatic and oversensitive. If your website is 99 per cent accessible and has one missing alternative text, the tool is likely to declare your whole website as having failed and show lots of red crosses on the screen. It may have even been the case that such an error was coded deliberately such as an image being used as a decorative element. If you've worked hard on addressing web accessibility issues and the tool says your website has failed, don't panic — manually go through the results and confirm if the tool has a point or if it's just being a bit sensitive.
4. Tools can take a while to be updated. While it's true that tools are updated to the latest standards and technologies, be aware that it can take a while for them to do so. Even now, some popular tools only support WCAG 1.0 or have WCAG 2.0 as 'beta' despite the standard being released four years ago. It's worth keeping an eye on exactly what the tool is validating.
5. No tool can replace manual checks and user testing. Many organisations turn to automated tools to avoid the need for user testing. By all means, use a tool to help address accessibility issues but to really get to the bottom of accessibility issues, feedback from people with disabilities can go a long way in ensuring that your website is accessible.

So while it's unlikely that the scenario discussed at the beginning of this column is likely to be eased any time soon, evaluation tools can be useful in providing a general accessibility overview as long as the dangers are kept in mind. With lots of work going on in the evaluation space at the W3C, I'm optimistic that in the coming years we'll see a more uniform and consistent way to evaluate website accessibility which will be a great thing for accessibility specialists and information and ICT alike.

Dr Scott Hollier represents Media Access Australia on the W3C Advisory Committee and publishes the W3C column monthly.