Accessibility Testing

2006-09-21T03:39:00.000-07:00

Accessibility Testing Software Compared
Steve Faulkner, Web Accessibility Consultant, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144. steven.faulkner@visionaustralia.org.au

Andrew Arch, Manager Online Accessibility Consulting, Vision Australia Foundation [HREF1], 454 Glenferrie Rd, Kooyong 3144. andrew.arch@visionaustralia.org.au

Contents: introduction | conformance | investigations | conclusion | references | detailed results

Abstract
Web accessibility for people with disabilities and other disadvantaged groups is becoming increasingly important for government and educational institutions as they try to meet their obligations under the Disability Discrimination Act and various policies and guidelines for online publishing. Business are also obligated under the DDA not to discriminate against people with disabilities as a result of their online activities.

A plethora of web accessibility testing tools have been released on to the market over the past eighteen months at prices ranging from a few hundred dollars to many thousands of dollars. This study looks at the ability of four of these testing tools to accurately assess the accessibility issues on a web site against the W3C Web Content Accessibility Guidelines 1.0 Priority 1 checkpoints. We conclude that they all have strengths and weaknesses and that none of them are able to identify all the accessibility issues. At this stage these tools can only aid accessibility testing, not provide a definitive assessment.

Introduction
With the increasing requirement for Government, education and business to provide accessible online services in order to provide access to all citizens including people with disabilities, many web managers are turning to the "robotic" tools to spider through their sites and tell them the problems and accessibility 'hot spots'. This has led to a veritable industry of software developers trying to solve this problem for those requiring "accessible" web sites, spured along with the recently enacted Section 508 [Section 508] law in the United State requring Federal Government agencies to build accessible web sites and generally to "buy accessible".

However, as we shall demonstrate in this paper, not all accessibility assessment software tools are created equal. Some overstate the problems while others understate the problems that exist on a web site. And they don't even do this consistently across the sixty five checkpoints of the Web Content Accessibility Guidelines 1.0 (WCAG) [Chisholm, et al ].

Web accessibility testing tools is a relatively new software area - most tools have been available for less than two years. Comparisons of the efficacy of the tools have not been undertaken as the market has matured. Graves [2001] in Government Computer News conducted one of the earliest comparisons between InFocus 508 and PageScreamer 2.3 noting the problems both products had with tables. Harrison [2002] and Harrison and O' Grady [2002] presented a more rigorous comparison of six analysis and repair tools noting the difficulty in isolating errors identified by some of the tools and the emphasis on US accessibility standards rather than the international WCAG ones.

Our aim has been to investigate the efficacy and accuracy of some of the available software applications used for web accessibility testing , in order to help assess the value of such tools in the broader process of ensuring the accessibility of web sites [Brewer & Letourneau].

How is conformance ascertained
What is accessibility?
An accessible web is available to people with disabilities, including those with:

Vision impairment (e.g. low vision or colour blindness) or vision loss affecting their ability to discern or see the screen
Physical impairment affecting their ability to use a mouse or keyboard
Hearing impairment or loss affecting their ability to discern or hear online audio
Cognitive impairments (e.g. dyslexia, ADD, learning difficulties, memory impairment) affecting their ability to comprehend or understand your site
Literacy impairments (e.g. low reading skills or English is not their first language) possibly affecting their ability to fully understand your site and its messages
Beneficiaries from an accessible web, however, are a much wider group than just people with disabilities and also include:

People with poor communications infrastructure, especially rural Australians
Older people and new users, often computer illiterate
People with old equipment (not capable of running the latest software)
People with "non-standard" equipment (e.g. WAP phones and PDA's)
People with restricted access environments (e.g. locked-down corporate desktops)
People with temporary impairments or who are coping with environmental distractions
How do we check for an accessible web site?
The Web Accessibility Initiative (WAI) outlines approaches for preliminary and conformance reviews of web sites [Brewer & Letourneau]. Both approaches recommend the use of 'accessibility evaluation tools' to identify some of the issues that occur on a web site. The WAI web site includes a large list of software tools to assist with conformance evaluations [Chisholm & Kasday]. These tools range from automated spidering tools such as the infamous Bobby [ Watchfire, 2003], to tools to assist manual evaluation such as The WAVE [WebAIM], to tools to assist assessment of specific issues such as colour blindness. Some of the automated accessibility assessment software tools also have options for HTML repair.

What role can automated tools play in assessing the accessibility of a web site?
WCAG 1.0 comprises 65 Checkpoints. Some of these are qualified with "Until user agents ..." and with the advances in browsers and assistive technology since 1999, some of these are no longer applicable - leaving us with 61 Checkpoints. Of these only 13 are clearly capable of being tested definitively, with another 27 that can be tested for the presence of the solution or potential problem, but not whether it has definitively been resolved satisfactorily. With intelligent algorithms many of the tools can narrow down the instances of potential issues that need manual checking, e.g. the use of "spacer" as the alt text for spacer.gif used to position elements on the page.

These automated tools are very good at identifying pages and lines of code that need to be manually checked for accessibility. Unfortunately, many people misuse these tools and place a "passed" (e.g. XYZ Approved) graphic on their site when the tool can not identify any specific accessibility issues, but the site has not been competently manually assessed for issues that are not software checkable.

So, automated software tools can:

check the syntax of the site's code
identify some actual accessibility problems
identify some potential problems
identify pages containing elements that may cause problems
search for known patterns that humans have listed
However, automated software tools cannot:

check for appropriate meaning
check for appropriate rendering (auditory, variety of visual)
The interpretation of the results from the automated tools requires assessors trained in accessibility techniques with an understanding of the technical and usability issues facing people with disabilities. A thorough understanding of accessibility is also required in order to competently assess the checkpoints that the automated tools cannot check such as consistent navigation, and appropriate writing and presentation style.

Investigation of efficacy of accessibility software
Choice of software
The choice of tools to review was based on a number of factors:

The software needed to have the ability check WCAG 1.0 checkpoints (some tools check only for US Section 508 problems). This decision was based on the applicability of the WCAG 1.0 to Australian disability regulations.
The least expensive desktop software products from each software producer. This decision was based upon the hypothesis that the more sophisticated versions of the testing software are built upon the same 'testing engines' and algorithms as the entry level products.
Potential users were considered more likely to purchase the less expensive products as Web Accessibility testing would not be a major priority for many organisations.
The availability of trial versions of the software products to be reviewed and the resources available to conduct the review.
We intend to expand the list of software products reviewed as software and resources become available

Table 1: Software Reviewed Software Vendor URL Cost
AccVerify 4.9 HiSoftware http://www.hisoftware.com US $495
Bobby 4.0.11 Watchfire http://www.watchfire.com US $99
InFocus 4.2 SSB Technologies http://www.ssbtechnologies.com/ US $1,795
PageScreamer 4.1 Crunchy Technologies http://www.crunchy.com/ US $1,495

1. A new version of Bobby has been released since the testing was conducted.

It is evident that there is quite a difference in cost across vendors for their entry level products. This difference in cost is partially reflected in the functionality of the software; some tools automatically fix certain problems, but the core functionality, the testing and reporting on WCAG 1.0 issues, is present in all the software reviewed.

Investigation methodology
Site used for testing
The site "The University of Antarctica" used for the review is a demonstration site developed by WebAIM, a non profit organisation whose stated goal "is to improve accessibility to online learning opportunities for all people".

The site contains examples of the many potential barriers to accessibility. The site is hosted by WebAIM at http://www.webaim.org/tutorials/uofa/.

The site consists of:

28 HTML documents
1 CSS file
16 GIF files
24 JPG files
1 SWF file (Flash)
1 Java file
13 AU files (audio)
1 MOV file (multimedia)
3 MPEG files (multimedia)
The site was chosen because it was built to demonstrate accessibility problems. The scope of the site is quite small and therefore instances of problems are easily quantified. Furthermore, the site was constructed using plain HTML files, there are no pages generated "on the fly' from a database, making the process of manual checking and quantification of the site a manageable task.

It was also reasoned that the site content and structure is relatively stable and therefore further testing of the site at a later time will still produce accurate comparison results. Further to this, a copy of the site will be stored at http://it-test.com.au/UOFA by Vision Australia Foundation to ensure the sites continuing integrity for testing purposes.

Process followed
Each of the products in the review were set to produce reports detailing issues in reference to the WCAG 1.0 Priority 1 Checkpoints. The reporting options if present were set to produce the standard reports. All reports were produced in HTML format.

AccVerify report comprised 60 HTML files:(Accverify report [zip file 1,128kb])

3 summary pages, comprising a listing of Priority 1 checkpoint errors and instances and Priority 1 visual checkpoints (needing human judgment) and instances. Graphical representation of this information is also presented along with a page listing the files that failed against any of the checkpoints along with a link to the associated detailed report and checklist
5 statistical pages listing some of the structural elements of the files tested, e.g. tables, forms, images, with links to the pages containing the elements
26 detailed report pages (1 for each file tested):
individual page summary and graphs of checkpoint errors and visual checkpoints
detailing of instances of specific issues with links to the associated checkpoints on the W3C web site.
divided into Priority 1, 2 and 3 issues
26 checkpoint pages (1 for each file tested):
listing of all WCAG 1.0 checkpoints with indication of whether the page passed/failed, not applicable, or needed visual checking for each checkpoint
short explanation of each checkpoint, paraphrased from WCAG 1.0 guidelines
divided into Priority 1, 2 and 3 issues
Bobby report comprised 72 HTML files: (Bobby report [zip file 183kb])

1 summary page consisting of a short description of all (possible) issues found and links to the site files where an occurrence of the issue was found
1 index page consisting of a list of all the files tested with links to their corresponding detailed report
35 detailed report pages (1 for each file tested):
detailing instances of specific issues with links to (locally stored) explanations of the issues and links to the associated checkpoints on the W3C web site
divided into Priority 1, 2 and 3 issues
further divided into issues either needing, or not needing, user checking to confirm their existence
35 (1 for each file tested) 'text only' versions of the files tested
InFocus report comprising 28 HTML files: (Infocus report [zip file 74kb])

3 summary/index pages:
a summary compliance report page, listing instances of and explaining each violation with links to the reports of "top 5 pages containing this violation".
a page with links to a detailed report for each page tested, also listing the number of checkpoint 'violations' found on each page
a page with links to a detailed report for each page tested ordered by (descending) total number of 'violations' found on each page
25 detailed 'compliance report' pages (1 for each file tested)
listing each violation found, the associated WCAG 1.0 checkpoint and the offending elements within the HTML code of the page.
PageScreamer report comprising 66 HTML files: (Pagescreamer report [zip file 256kb])

1 page containing a copy the WCAG 1.0 guidelines from the W3C web site
1 'detail' page listing every HTML element that triggered a checkpoint violation, the elements' URL location and line number
1 verification summary page for the site
containing a graph showing the number of 'tags' in violation of each checkpoint
a table listing all the checkpoints with links to the related sections of the locally stored copy of the WCAG guidelines.
numbers of Compliance violation instances and whether the site passed/failed the checkpoint or the checkpoint needed further verification
1 file containing a text description of the graph
27 verification summary pages (one for each URL found) containing:
containing a table listing all the checkpoints with links to the related sections of the locally stored copy of the WCAG guidelines.
numbers of compliance violation instances and whether the site passed/failed the checkpoint or the checkpoint needed further verification
35 pages containing lists of links to URL's (with links to associated report files) containing instances of a violation of a particular checkpoint (1 page per checkpoint)
Overview of results
A significant measure of the software tools and their ability to report on the accessibility problems of a site is the ability to find all the URL's on the target site. Some of the apparent discrepancies between software of the URL count could be apportioned to the software only listing the URL's of files that contain errors. But in this situation all 28 HTML files contained at least one instance of non conformance with a WCAG 1.0 checkpoint.

Table 2: HTML files found AccVerify Bobby InFocus PageScreamer Manual Check
26 20 25 27 28

AccVerify
Failed to find any instances of the MPEG (3) (multimedia) or AU (13) (audio) files
Failed to find files (2) that were the targets of forms
Bobby
Identified instances of audio and multimedia files, although it failed to report some instances.
Failed to find a file linked via meta based redirect
Tested and reported twice on the 'Home' page [index.html]
Failed to find a file linked via an image map based link
Failed to find files (2) that were the targets of forms
InFocus
Failed to find files (3) that were the targets of forms
PageScreamer
Tested and reported twice on the 'Home' page [index.html]
Failed to find files (2) that were the targets of forms
Found instances of the MPEG (3) (multimedia) or AU (13) (audio) files but passed site on checkpoints relating to multimedia without verification.
Accuracy and Definitive nature of results
Another significant measure of the products efficacy is its ability to produce both accurate and definitive results without the need for further human interpretation.

Table 3: Status of (15 priority 1) checkpoints tested AccVerify Bobby InFocus PageScreamer Manual Check
failed 2 2 5 3 11
passed 2 n/a 3 - 4 4 4
Human
intervention (%) 11 (73%) 11 (73%) 8 (53%) 6 (40%) n/a
Not reported 0 2 1 2 1 2 2 n/a

Bobby and InFocus did not report upon issues that could be checked by the software and found not applicable e.g. they found no server-side image maps therefore did not report about issues concerning client side image maps
PageScreamer failed to provide any information in the report about 2 Priority 1 checkpoints (4.1, 14.1).
Bobby only reported on those checkpoints that it found the site to be in breach of.
PageScreamer incorrectly reported the site having passed 2 checkpoints (1.3, 1.4) that the manual check revealed to be fails.
Graphical representation of data from Table 3

Table 4: Totals of reported potential/actual failures of (15 priority 1) checkpoints tested AccVerify 1 Bobby 2 InFocus3 PageScreamer4 Manual Check
failed 2 2 5 3 11
Potentials Failures
(human Intervention)
11 11 8 6 n/a
Totals 13 13 13 9 11

AccVerify overstated total potential failures in relation to actual failures
Bobby overstated total potential failures in relation to actual failures
InFocus overstated total potential failures in relation to actual failures
PageScreamer defined the least checkpoints as 'potential failures', but understated total potential failures in relation to actual failures
Graphical of representation of data from Table 4.

Over reporting
'Over reporting of instances of potential checkpoint failures was a common feature of all the products reviewed.

Table 5: Examples of reported Instances of potential/actual checkpoints failures AccVerify Bobby InFocus PageScreamer Manual Check
2.1 - Ensure that all information conveyed with color is also available without color 26 47 25 220 1
5.1 - For data tables, identify row and column headers. 11 12 1 164 1
5.2 - For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells. 11 13 4 464 1

Discussion
While Bobby was the most successful tool in identifying multimedia and audio files it had the most problems identifying HTML files linked via the META and IMAGE MAP elements as well as those files that were the targets of FORM elements. The inability to identify some of the targets of Forms was a common defect among all the software.

The product's ability to give a quantitative answer as to whether the site passed or failed against 15 WCAG 1.0 Priority 1 checkpoints varied from 60% for Pagescreamer to 28% for AccVerify, though it should be noted that the Pagescreamer quantitative results produced 2 false passes. The results highlight that the software tools either did not produce a quantitative report or could not produce an accurate report on the status of the site in relation to the majority of the Priority 1 checkpoints tested. For approximately 10 of the 15 Priority 1 checkpoints tested the reports inform us that the checkpoints need a 'visual', 'manual' or 'user' check. Furthermore, although along with the instructions to do a 'manual' check the reports detail potential instances of checkpoint violations (Table 5), which should be helpful in tracking down issues, when comparing the number of potential instances reported against the actual occurrence, it is evident that none of the tools does a very good job at identifying potential errors. All of the products over-reported potential checkpoint errors/violations. A number of checkpoints were detailed as potential errors on every page by some software tool reports.

All of the products produce a report that upon initial consideration may appear as a detailed analysis of the accessibility problems found by the product on the site tested. Upon closer examination it is revealed that the software tools fail at the initial hurdle of correctly identifying the files to be tested. Furthermore, many of the accessibility issues that may occur on the site are not within the reach of the 'mechanical' rules based analysis that these products undertake. In an attempt to ensure that issues are not missed by the software, the reports tend to overstate the occurrence of potential problems, up to the point where a potential instance of a checkpoint violation is flagged for every file checked, thus undermining even the heuristic values of the report.

Conclusion
The research into the automated accessibility tools reported here and conducted at Vision Australia Foundation indicates that the rule of "caveat emptor" applies as equally to the field of accessibility testing tools as it does to buying a used car.

All of the tools tested have advantages over free online testing tools [Steven Faulkner], the main adavantage being their ability check a whole site rather than a limited number of pages. Also files do not have to be published to the web before they can be tested, the user has greater control of what rules are applied when testing, the style and formatting of reports, and in some cases the software will automatically correct problems found.

None of the tools evaluated, as expected, were able to identify all the HTML files and associated multimedia files that needed to be tested for accessibility on the site.

All the software tools evaluated will assist the accessibility quality assurance process, however none of them will replace evaluation by informed humans. The user needs to be aware of their limitations and needs a strong understanding of accessibility issues and the implications for people with disabilities in order to interpret the reports and the accessibility issues or potential issues flagged by the software tools.

We hope the analysis reported here will aid web site quality assurance managers in their choice of web accessibility testing software, and in their understanding of the limitations that automated tools have in the broader process of ensuring accessible web sites.

References
Chisholm, W. et al (Eds) 1999, Web Content Accessibility Guidelines 1.0, World Wide Web Consortium. [HREF2]

Brewer, J & Letourneau, C. (Eds) 2002, Evaluating Web Sites for Accessibility, World Wide Web Consortium. [HREF3]

Chisholm, W & Kasday, L (Eds) 2002, Evaluation, Repair, and Transformation Tools for Web Content Accessibility, World Wide Web Consortium. [HREF4]

Watchfire, 2003, Welcome to Bobby.[HREF5]

WebAIM, undated, WAVE 3.0 Accessibility Tool.[HREF6]

Section 508: http://www.usdoj.gov/crt/508/508law.html [HREF7]

Graves, Steve 2001, Check sites for 508 with audit-edit tools, Government Computer News [HREF8]

Harrison, Laurie 2002, Web Accessibility Validation and Repair - Which Tool and Why? (introduction), Center On Disabilities Technology And Persons With Disabilities Conference 2002 (CSUN-2002) [HREF9]

Harrison, L & O'Grady, L 2002, Web Accessibility Validation and Repair: Which Tool and Why? (analysis), ATRC, University of Toronto.[HREF10]

Steven Faulkner, Vision Australia Foundation, 2003, Free Web Development and Accessibility Tools. [HREF11]

Hypertext References
HREF1
http://www.visionaustralia.org.au/webaccessibility/
HREF2
http://www.w3.org/TR/WCAG10/
HREF3
http://www.w3.org/WAI/eval/
HREF4
http://www.w3.org/WAI/ER/existingtools.html
HREF5
http://bobby.watchfire.com/bobby/html/en/index.jsp
HREF6
http://www.wave.webaim.org:8081/wave/index.jsp
HREF7
http://www.usdoj.gov/crt/508/508law.html
HREF8
http://www.gcn.com/20_23/reviews/16783-1.html
HREF9
http://www.csun.edu/cod/conf/2002/proceedings/279.htm
HREF10
http://snow.utoronto.ca/access/evaltoolreview/validation.html
HREF11
http://www.visionaustralia.org.au/webaccessibility/workshops/references.html#acheck

Appendix - Table of results
Priority 1 Checkpoints and Indication of software report on each checkpoint In General (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.1 Provide a text equivalent for every non-text element (e.g., via "alt", "longdesc", or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIF's), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video. fail fail fail fail fail
2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup. fail visual user check manual check verify
4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions). fail visual user check manual check not reported
6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document. fail visual user check manual check verify
6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes. fail visual user check manual check verify
7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker. pass visual user check manual check verify
14.1 Use the clearest and simplest language appropriate for a site's content. pass visual user check manual check not reported
And if you use images and image maps (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.2 Provide redundant text links for each active region of a server-side image map. pass pass not reported not reported pass
9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape. pass pass not reported not reported pass
And if you use tables (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
5.1 For data tables, identify row and column headers. fail visual user check fail verify
5.2 For data tables that have two or more logical levels of row or column headers, use markup to associate data cells and header cells. fail visual user check fail verify
And if you use frames (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
12.1 Title each frame to facilitate frame identification and navigation. fail fail fail fail fail
And if you use applets and scripts (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
6.3 Ensure that pages are usable when scripts, applets, or other programmatic objects are turned off or not supported. If this is not possible, provide equivalent information on an alternative accessible page. fail visual user check manual check fail
And if you use multimedia (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
1.3 Until user agents can automatically read aloud the text equivalent of a visual track, provide an auditory description of the important information of the visual track of a multimedia presentation. fail visual user check fail pass
1.4 For any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation. fail visual user check manual check pass
And if all else fails (Priority 1) Manual Check AccVerify Bobby InFocus Page Screamer
11.4 If, after best efforts, you cannot create an accessible page, provide a link to an alternative page that uses W3C technologies, is accessible, has equivalent information (or functionality), and is updated as often as the inaccessible (original) page. not tested

Copyright
Vision Australia Foundation, © 2000. The authors assign to Southern Cross University and other educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to Southern Cross University to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the conference papers and for the document to be published on mirrors on the World Wide Web.

bobyjos

Accessibility Testing

Test 2

Test Posting