upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility

Panayiotis Koutsabasis, Evangelos Vlachogiannis, and Jenny S. Darzentas

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 157 - 171

Article Contents


The method used to evaluate the accessibility of e-publishing web sites includes the following two basic steps that are a part of any comprehensive web accessibility evaluation process.

For the purposes of the accessibility evaluation of scientific e-commerce web sites, the following decisions were made:

A practical way to perform these heuristics is to use the Mozilla Firefox browser with the web developer toolbar, which enables turning off/on various web design elements to ensure well-formed design and accessibility.

The aforementioned approach to accessibility evaluation is sufficient in order to give an overview of accessibility problems and indicate the major areas that need improvement. This approach integrates basic steps that can be applied in a time-saving way by evaluators who do not need to be proficient with web development technology. However, this approach is not a complete account of the steps required for a complete web accessibility evaluation process, which is presented later in this paper.

Summary of Accessibility Problems Found

Table 1 shows the accessibility problems of scientific e-publishing homepages identified by the automated evaluation tools used. The results of automated evaluation can vary between web accessibility tools due to their differences in the implementation of automated checks. In particular, Imergo seems to make a more detailed automated web accessibility check in comparison to Cynthia Says with respect to some aspects of WCAG double-A and triple-A priorities.

Table 1. Accessibility problems of scientific e-publishing homepages identified by automated evaluation tools (Cynthia Says, HiSoftware & Imergo, Fraunhofer, FIT).

Table 1

The overall level of web accessibility for the scientific e-publishing homepages is not satisfactory. All 10 homepages examined had some accessibility problems, however these vary. Some present few accessibility problems, e.g., H2 (Homepage 2) presents five accessibility problems in total when evaluated by the Cynthia Says tool and four problems when evaluated by Imergo. On the other hand, the automated web evaluation for some of scientific publishing homepages is quite disappointing. For example, H1 presents a total of 115 problems when evaluated by Cynthia Says and 246 when evaluated by Imergo. The homepages that present many web accessibility problems are totally inaccessible and would need considerable redesign effort to reach a good level of accessibility. Overall, three sites have quite a large number of problems (ranging from 24 to 106!), five homepages have a few accessibility problems, while two homepages are clean at the single-A level. Similarly, the double-A and triple-A levels of accessibility evaluation also reveal a considerable number of problems for most web sites.

As part of the second step of the method, Table 2 provides an overview of accessibility problems identified by human judgment based on the heuristics discussed above.

Table 2. Accessibility problems of scientific e-publishing homepages identified by human judgment found after applying simple heuristics.

Table 2

The accessibility problems that were found in scientific publishing homepages include the following:

Again, it is evident that there are significant accessibility problems in the scientific e-publishing homepages and that not one homepage was judged as adequately designed to meet all the criteria offered by the heuristics. Six out of ten e-publishing web homepages have at least one major accessibility problem. Also, all sites have minor accessibility problems. These problems are different for each homepage and mainly concern information organisation, navigation, visibility, and user control.

A comprehensive accessibility evaluation cannot stay at the level of homepages but must also include other sections of the scientific e-publishing web sites. However, given that the homepages of scientific e-publishing web sites are not accessible, it is quite safe to assume that the rest of the application will include at least the same types of problems. This example demonstrates that even the contemporary scientific publishing sites present a wide range of accessibility problems. Scientific publishing homepages will reveal several problems when accessed in constrained situations, such as, access from mobile devices, black and white print-outs, access from voice browsers, access from assistive technologies used by people with motor disabilities, and so on.

Previous | Next