Abstract

The tutorial will explain how to set usability performance requirements based on effectiveness, efficiency and satisfaction, which can be measured once a prototype is available. It will give practical examples of how the approach has been implemented in industry.

Learning objectives

Participants will learn simple techniques that can be used to specify usability requirements based on:

• Identifying the range of contexts in which the product or system will be used
• Estimating task times for important scenarios of use
• Setting accuracy and completion criteria for important tasks
• Establishing satisfaction requirements
• Using the Common Industry Format to document usability requirements
• Identifying the key design issues that will impact on usability

By the end of the day participants will have sufficient knowledge to know how to introduce usability performance and satisfaction requirements into their own organisation. Usability performance requirements link usability to business requirements for productivity and accurate work. The tutorial will also show how design issues can be identified that can help reduce the likelihood of usability errors and contribute to achieving the usability requirements.

Audience

The tutorial is intended for anyone wishing to gain practical experience of specifying usability requirements. Some previous experience of usability is desirable, but not essential, as the approach taken will be business-oriented. The tutorial is not aimed at researchers, and some aspects of the methods may be familiar to experienced usability practitioners.

Origins

The tutorial incorporates material from previous tutorials on user centred design and usability requirements given at international conferences (including UPA), combined with case study material from training a large organisation to implement these methods. The user centred design tutorial was most recently given at UPA 2002, where the average answer to the question: "This tutorial should be offered again at next year's conference" on a scale of 1 to 5 was 4.75.

How tutorial will be conducted

The tutorial will be illustrated by case studies, and will include both class and group exercises to apply the methods. The tutorial will be supported by reference materials and defined procedures.

Detailed description of material covered by tutorial and a schedule of events with time allocation

The following topics will be covered:

1. An introduction to the user-centred design process described in ISO 13407, and how other related standards (ISO 9241-11: Guidance on usability, ISO/IEC 9126-4: Quality in use metrics and ISO/IEC WD 25030: Quality requirements) support specification and measurement of usability. Background to the case studies.

   [Trials of setting and evaluating usability requirements based on the Common Industry Format were made in four organisations coordinated by the author. The author has also trained business managers in a government department to apply these methods as a routine part of systems development.]

2. The importance of specifying the context of use, and how to use the Usability Context Analysis questionnaire to identify the range of intended users, tasks and environments, and to identify critical design issues.

   [Usability Context Analysis is a structured method to assist with user, task and environmental analysis, which is particularly useful for non-usability specialists and beginners.]

3. How to write meaningful task scenarios and estimate the associated task times.

   [How to write user-oriented scenarios, how to choose the right scenarios, how to estimate requirements based on information from existing systems, paper prototyping and step-by-step analysis.]

4. A structured method for estimating the extent to which user errors will reduce the accuracy and completeness of task output, and for assessing the magnitude of the resulting business costs. Identifying associated design issues.

   [From a business perspective it is not the usability errors themselves that matter, it is the time and effort taken to correct detected errors and the business consequences of undetected errors.]

5. Specifying satisfaction requirements using psychometric questionnaires.

   [Established questionnaires such as SUMI or QUIS can be used to baseline satisfaction requirements.]

6. Documenting usability requirements for internal use, or more formally using the Common Industry Format.

   [Internal documentation should be used to communicate information to developers and to provide a basis for subsequent usability testing. The Common Industry Format for usability test reports has recently been made a US standard. The same format can be used to specify requirements as part of a contract.]

7. Integrating these methods with existing processes.

   [Examples of how these methods can be integrated with existing design and development processes, focussed on the interests of the audience.]

Annex: Setting usability performance requirements - rationale

This tutorial is about the value of setting usability requirements that can subsequently be tested with summative evaluation.

The principle is not new. Much early usability work used summative methods, but was not always supported by other user centred design activities. It therefore gained the reputation for being an expensive way to identify problems when it was too late to fix them! So the emphasis moved to formative evaluation (so-called “discount” usability methods) that could be used earlier in development.

As usability work has matured to provide cost-effective techniques for use during design and development, there is increasing value in introducing the same controls as used in other areas of software engineering: defining clear requirements, and evaluating whether the requirements have been met. This principle has recently been formalised in a series of international standards for software quality. They all emphasise the importance of expressing user requirements in the form of goals for effectiveness, efficiency and satisfaction.

Defining usability performance (and satisfaction) requirements is particularly important because of their close relationship to business objectives. Task outputs that are inaccurate or incomplete cost businesses money to deal with the consequences. Tasks that take too long to complete cost time which in most situations also means money. The consequences are most clearly seen in the development of in-house systems. For example, lack of user performance requirements was a fundamental reason for the expensive costs and delays incurred when new passport issuing software developed by Siemens substantially increased the time it took operators to issue passports in the UK.

Concerns about uncontrolled costs were a major motivation for the development of the Common Industry Format, an initiative supported by major corporations in the US. The CIF provides a means for a supplier to report usability results to a customer, so that usability can be considered as one of the purchase criteria.

An international standard is currently being developed for the usability of consumer products, which aims to make information on the successful task completion rate with consumer products available to the public.

The principles on which this tutorial is based are not new. But it is only recently that they have been formalised in standards, and that it has become realistic to expect industry to specify and measure these requirements. The tutorial explains how to do this based on practical experience.

Currently these requirements are rarely specified, but doing so could provide immense benefits.

Instructor's Biography

Dr Nigel Bevan is Research Manager at Serco Usability Services. Nigel coordinated European-funded projects that developed and trialled the methods, and he has subsequently applied them commercially. He is currently managing the UsabilityNet project that has established a web site of usability resources. Nigel provides consultancy in usability and user centred design. He is active in several international standards groups, and contributed to development of the Common Industry Format. Nigel has given tutorials on usability and user centred design at several international conferences including the UPA.