upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility

Panayiotis Koutsabasis, Evangelos Vlachogiannis, and Jenny S. Darzentas

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 157 - 171

Article Contents


Steps Towards a Practical Methodology for Web Accessibility Evaluation and Maintenance

The example of web accessibility evaluation is an important but rather small part of the practical measures that need to be taken up to ensure web accessibility. To try to work out a practical methodology for evaluating web accessibility is very important for at least two reasons. Firstly, web accessibility is a huge, diffuse and complicated more or less theoretical field of human-computer interaction so any attempt to come up with a practical design tool and advice is valuable. Secondly legislation regulating web accessibility, especially concerning official government and federal web sites and services, are now in force and are being extended into the private corporate domain. These laws are pitched at a general and abstract level, so concrete advice and practical guidelines are urgently required by anyone who designs for the web if they are to ensure that their designs comply.

Therefore we propose that a holistic approach to the management of web accessibility evaluation should include (a) identification of user requirements and set up of accessibility goals, (b) web accessibility evaluation and redesign process, and (c) establishment and follow-up of an accessibility policy.

Identification of User Requirements and Set Up of Accessibility Goals

User accessibility requirements can be identified when accessibility is included in the goals of a user-centred approach for design and evaluation. The rationale behind important work on web accessibility is that of “universal design” or “design-for-all” that includes all users without discriminating particular (types of) disabilities. On the other hand, there may be particular user accessibility requirements to be included in a web development/evaluation project that are related to specific target user groups. The general accessibility requirements of particular user groups have been reported in literature (e.g., see WebAIM, Considering the User Perspective: A Summary of Design Issues, http://www.webaim.org/articles/userperspective); however, there may be particular requirements for specific groups or access contexts that need to be identified.

When there is a need to incorporate particular requirements for specific user groups (e.g., deaf users), it is critical to remember the universal design principle and thus ensure that the web site will conform to at least a minimum level of web accessibility specifications. At present the WAI web accessibility specifications allow for levels of conformance to the guidelines. Thus, it is possible for web designers to achieve accessibility conformance at the single-A level (lowest level) and then specialise their designs for particular user groups. However, it needs to be noted that according to W3C.WAI WCAG2.0: “When a Web page is one of a series of Web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all Web pages in the process conform at the specified level or better” (W3C, 2008, Conformance Requirements, number 3). Therefore, accessibility goals may be set to meet the following:

Besides the requirements that arise from the users themselves, the web site owners can decide upon a hierarchy of services and concentrate in the first instance on making the top level accessible. For example, in the case of scientific electronic publishing a primary (top level) use might be to download a paper in a journal, whereas other services, such as the alerting service, may be given a lesser priority and be dealt with at a later stage. This does not mean that these services should not be made accessible, because by leaving them inaccessible it is not possible to claim conformance with the W3C, but more importantly it creates a two-tier user base: those who can use everything and those who can only use certain services. However for practical purposes, when faced with making a whole set of services accessible, it is reasonable to prioritise.

Web Accessibility Evaluation and Redesign Process

As the accessibility evaluation example has shown, a rapid assessment of web accessibility is possible by using free accessibility tools (automated tools that measure technical accessibility conformance to guidelines) and applying simple heuristics (manual, expert-based accessibility inspection). The use of the automated tools and the manual inspection that uses heuristics are next described as stage two of the proposed methodology. Finally, a holistic approach to accessibility evaluation and redesign also requires another important element in the process: that of user involvement through user testing. These phases of web accessibility evaluation have been also identified by other researchers such as Arch (2008), who identifies three types of web accessibility testing: automated testing, which is carried out by software tools; manual testing, which is carried out by human evaluators; and user testing, which is carried out by end-users.

Tool-based accessibility conformance to guidelines

Accessibility evaluation tools scan the source code of a web page using interpretations of either WCAG or the United States Rehabilitation Act Section 508 standard. The use of these tools is the first step for accessibility evaluation because they can quickly identify accessibility problems that can be identified at the level of the source code of a web page and produce reports with accessibility errors and warnings. These tools save the designer from the task of inspecting source code for the evaluation of accessibility and provide a first account of accessibility problems. However they cannot provide a complete account of accessibility problems mainly because accessibility is not a solely technical issue, but primarily requires human judgment. According to WebAIM (2010) of the combined 65 checkpoints in WCAG 1.0 Priority 1 through Priority 3, only 19 can be partially evaluated automatically.

A major problem for accessibility tools is that the vast majority of these tools are designed for fast evaluations of single web pages. There are no public accessibility tools that can reside on the web server, constantly monitor the web site, and are capable of crawling an entire web site to identify accessibility problems. Currently, research on the design of new generation accessibility tools attempts to address these concerns such as the MAGENTA tool (Leporini, Paterno, & Scorcia, 2006) and the BenToWeb benchmarking tools that will include the aforementioned capabilities (Herramhof et al., 2006).

Expert (manual) accessibility inspection

Manual evaluation includes a number of steps that must be followed by a designer to check the accessibility of web pages according to guidelines. These steps are another essential task of accessibility evaluation that can assess accessibility in terms of the aspects that require human judgment. Such aspects include, for example, that alternative text for images substantially describes the meaning of an image in textual form, in case this is needed (i.e., when the image conveys information and is not used for other purposes such as decoration), and that the use of colours promotes accessibility of text and images if viewed in a constrained context of use (i.e., when printed by a black and white printer).

Expert inspections of accessibility can identify a considerable number of problems that are not possible to find by using automated accessibility tools alone. Typical problems in this respect were demonstrated in the evaluation example presented in previous section. Typical inspections of web accessibility include the following:

The expertise required to conduct an accessibility evaluation should be wide-ranging, including both organisational and technical skills. According to the W3C.WAI an accessibility evaluator should have “an understanding of web technologies, evaluation tools, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and accessibility guidelines and techniques” (2010, Evaluating Accessibility/Combined Expertise page, Recommended Expertise section).

User testing of accessibility

The involvement of users with disabilities in accessibility testing is an important aspect of accessibility evaluation. As Paddison and Engefield (2004) point out “it is not enough to follow accessible guidelines and make the appropriate technical accessibility changes... people with special accessibility needs should be considered as a distinct user profile with their own requirements, within a user-centred design process” (p. 508). It has been observed over and over that user testing with people with disabilities contributes to a better understanding of accessibility issues by all people involved, and especially web developers (Foreign & Commonwealth Office, 2010). For example, having web developers see people with disabilities accessing a web page with a voice browser helps them to identify related accessibility problems that their web site may have, such as the inappropriateness or absence of alternative text, the ordering of controls in a form, etc.

However, a user-centred accessibility evaluation will not be effective unless the site is already at a minimum level of accessibility. Even so, the inclusion of users in the accessibility evaluation process can also identify various usability, as opposed to basic accessibility, problems. Analytic methods and guidelines for involving users in accessibility evaluation include the work of Gappa and Nordbrock (2004) and Petrie, Hamilton, King, and Paven (2006).

Web Accessibility Policy

In order to reach and maintain web accessibility, web site owners need to establish a web accessibility policy that will be applied during the design, development, and operation of their web site. The accessibility policy needs to go beyond an “accessibility statement” that typically declares the specifications that web sites conform to. The policy needs to define systematic procedures, checks and tools for content updating. It has been many times observed that web sites that were accessible when first released gradually lost their accessibility when the content was updated. As an example, educators of an accessible e-learning platform or users of an accessible web log need to add content that conforms to web accessibility; this can be achieved if people are aware of accessibility issues and are able to use tools to check the content accessibility when it is inserted into the site. This is particularly important for web 2.0 applications that allow for user generated content.

Currently, web accessibility statements have been established for a number of, mainly academic and governmental, web sites and in a few e-commerce systems that are designed with the participation of organizations of people with disabilities, such as the RNIB (e.g., Gladstone, Rundle, & Alexander, 2002). An important issue for the design of the web accessibility policy is what standards, guidelines, methods, and processes should be identified from related work. In spite of the fact that there is some work in respect of accessibility policy (for a review, see Gulliksen, Harker, & Vanderheiden, 2004), we are still some way away from the recognition of a systematic accessibility policy that is implementable to the level of content generation by end users.

Previous | Next