Workshops

An expert inspection method that has been shown to improve evaluators' performance in Heuristic Evaluations is shared with participants. The aim is to generate tighter ‘profiles' of problems typically found in different kinds of applications enabling evaluators to focus on a small sample of specific, rather than generic, heuristics.

WORKSHOP DESCRIPTION

The problem

To date comparative studies have not convinced the HCI community that results from usability evaluations are reliable. Evaluations of the same products by different usability teams found different problems (Molich et al., 1999). Discount methods have the same problem. With the Heuristic Evaluation (HE) method, for example, overlaps between evaluators' results are often as low as 10% (Molich & Robin, 2003). However, HE has more problems than user testing. An analysis by Bailey (2001) revealed that only 36% of all the problems identified were true usability problems while 43% were false alarms and 21% of known problems were missed. It is thus questionable whether discount methods are safe enough to justify their low cost (Cockton & Woolrych, 2002).

Cockton & Woolrych (2002) suggest that we “improve all HCI methods so that discount methods are less discounted and ‘full strength' methods can be applied in more contexts”. Despite the somewhat negative publicity that HE has received lately, usability practitioners are still using HE more often than any other method. This suggests a pressing need for good discount methods. As long as ‘full strength' methods are still not reliable, time-consuming, and too costly for companies with small budgets, practitioners will continue to use discount methods. Thus, it is worthwhile to take a more pragmatic approach to do our best to improve existing discount methods, in particular, HE that seems to be so much used by practitioners.

Background research

Recently, one of us identified a ‘profile', which consists of usability problem areas associated with the complex application that she evaluated employing a naturalistic inquiry approach. From this research, she pioneered the concept of ‘usability problems profile', which was defined as common problem areas found for the same type of products. This concept was then used to extend and improve the reliability of HE results and ease of use of the method. She and a colleague introduced the HE-Plus method, in which evaluators are given a list of problem areas that constitute a ‘usability problems profile' to be taken into consideration when applying the list of heuristics used. They tested the method in an experimental study that compared heuristic evaluation (HE) to HE-Plus. In the study, participants evaluated an online shopping website. Results from the study showed that HE-Plus yielded significantly more overlaps than HE. The data obtained also allowed them to correct the original ‘profile' for websites that they used in the experiment, which they approximated from available information in the literature. Later, a second study on HE-Plus was carried out with the same type of websites, using the ‘profile' derived from the data of the first study. The results of this second study confirmed the findings of the first study. Firstly, HE-Plus outperformed HE. Secondly, the sets of problem areas making up the ‘profile' that was derived from the data in both studies were same. Furthermore, contrary to the results of the first study, they found that the procedure to carry out HE-Plus had been improved in the second study: evaluators rated HE-Plus as being easier than HE.

These two studies show that we are heading in the right direction of making heuristic evaluation a better discount method. If we knew what constitutes ‘profiles' for different product types, we could tighten the HE procedure and get better evaluation results because ‘profiles' apparently help evaluators to remain focused. It is possible to derive ‘profiles' from companies' old usability evaluation data and customers' feedback, for example. In the future, we may compile a ‘profile bank', a database of problem areas for different types of applications, so that evaluators can choose an appropriate profile for the evaluation.

Until the ideal ‘full strength' methods such as user testing become feasible for practitioners, discount methods should be improved. We wish to extend the course of this research further to include inputs from usability practitioners because they are the ones who would benefit most from a more reliable inspection method.

The aims of the workshop

•  To instigate an industrial collaboration in a research project that aims to improve the reliability of discount evaluation methods.

•  To test if ‘profiles' help non-usability specialists such as designers and developers conduct heuristic evaluations with comparable outcomes to those of experts.

•  To debate on the roles of inspection methods and user tests in professional practice.

What would participants take home?

•  A version of heuristic evaluations that gives more reliable results than the original method.

•  Non-usability experts (designers/developers) who participate in the pre-workshop activities will have had experience using HE-Plus and so can use it in future. This will be help reducing their companies' development time and cost.

•  For usability experts, ‘profiles' and appropriate heuristics that we hope to generate as a result of this workshop could help them design better user tests or ‘cut corners' effectively and reliably while doing heuristic evaluations.

What is expected of participants?

Workshop participants are expected to perform a usability test or a heuristic evaluation using the traditional method or the one that will be discussed in this workshop (HE-Plus). The organizers will assign both the product to be evaluated and the evaluation-type as far as possible according to participants' preference, resource, and experience as indicated in their position paper.

PARTICIPANT SELECTION CRITERIA

Participant selection will be based on the criteria below and their position paper. Participants may be usability practitioners, designers, or developers who

• Are keen to improve HCI evaluation tools.
• Are frustrated with the generality of current versions of heuristic evaluations.
• Conduct heuristic evaluations often although they are not confident in the results.
• Like to share their experience on the issue that discount methods are too risky to justify their low costs.

We invite position papers on one or more of the following themes:

1.  The roles of heuristic evaluation (HE) and user testing in professional practice.

• What should they be?
• Why and when do you use the methods?
• How often do you use HE in your professional work comparing to user testing?

2. Experience report or reflection on cases when you used HE even though user testing would be a better alternative.

3. Issues on reliability and validity of usability evaluation methods (UEMs) in real world practice.

4. “Different evaluators/teams finding different problems in user testing or heuristic evaluation”

• Does it bother you at all?
• Why and why not?
• What should usability practitioners do about it?
• How might evaluation results from HE and user testing be improved?

The position paper will also include

• A statement of how the participant meets the selection criteria.
• Agreement to carry out one of the following methods: user testing; HE; or HE-Plus
• Participant's top two choices for the above three methods

Click here to link to template for position paper

Click here to link to sample position paper

APPLYING TO PARTICIPATE IN THIS WORKSHOP

The workshop is a closed session, limited to 20 participants. Admission to a workshop requires an approved position paper from you addressing the issues suggested by the organizers. Please send your position paper (which should be roughly 1 to 2 pages) to Gitte Lindgaard, Gitte_Lindgaard@carleton.ca. Position papers deadline is March 26. Notification of acceptance/rejection will be sent to you by April 9. Papers received after March 26 will be evaluated at the facilitator's discretion. If you want to register early for UPA and have not been confirmed of your acceptance, you may register for the rest of the conference and add the workshop fee later.

PRE-WORKSHOP PARTICIPANT ACTIVITIES

Instruction for testing/evaluating sent to participants will include:

• Information about the application to be evaluated.
• Detail about the method participants will use.
• A set of user tasks.
• Detailed test/evaluation instructions.
• Deadlines.
• The content and format of an informal report.

PRE-WORKSHOP FACILITATION ACTIVITIES

1. We will upload participants' position papers and general information about the workshop on the workshop website at www.interactingwithcomputers.com.
2. We will analyze the data from participants' problem lists to prepare for presentation of results at the workshop.
3. We will produce handouts of the results to be given to participants at the workshop.

POST-CONFERENCE DISSEMINATION OF RESULTS

The workshop results will be presented during the conference as a poster. We will make the outcomes of the workshop available to our participants on our website, produce a report for publication on the UPA Voice, and also plan to submit a full paper for a journal or, if appropriate, act as guest editors of a special issue of a HCI journal.

POST-CONFERENCE ACTIVITIES

We will maintain the workshop website at www.interactingwithcomputers.com. The aim is to make available to public the information about the new method, which results from the workshop.

FACILITATORS

Gitte Lindgaard

Professor & Chair, User-Centred Design, Director Human Oriented Technology Lab (HOTLab)

Carleton University, Canada

Gitte has been active in HCI as a researcher, a consultant, an academic, teacher, and practitioner for the past 25 years in four continents. She holds the first PhD in HCI from Australia, the gist of which was an exploration of a variety of evaluation methods, including heuristic evaluation, for designing help systems. As head of Human Factors in the Australian national telecommunications carrier, she pioneered a range of investigative and evaluative methods that were then integrated into the software development process. Gitte is a fellow of the Ergonomics Society of Australia, a member of HFES, ACM SIGCHI, CIPS (Canadian Information Processing Society), and other professional organizations. Currently, she holds the National Science & Engineering Research Council/Nortel/Mitel/CITO/OCRI Chair in User-Centred Design and is Director of the Human Oriented Technology Lab (HOTLab) at Carleton University in Ottawa, Canada.

Jarinee Chattratichart

Senior Lecturer

Department of Computing, Communications Technology and Mathematics

London Metropolitan University, UK

Jarinee is a Senior Lecturer in User Interface Design and MSc IT Usability at London Metropolitan University. She holds a Ph. D. degree in HCI from Brunel University. Her academic background includes HCI, Computer Science, Chemical Engineering, and Environmental Engineering. Her favorite subjects are UCD methodology and research methods. Jarinee is also an enthusiastic researcher in a wide range of research topics, which includes discount methods in usability evaluation and requirements gathering, m-commerce, web usability, and usability of visual programming languages.

Thyra Rauch

Customer Research Architect

IBM, U.S.A.

Thyra, who has worked at IBM for seventeen years, currently works for IBM's Data Management team working on information integration and search solutions. She has been instrumental in validating product and UI direction, user roles, and role/task pairings across a number of different industries for multiple product solutions. She has been using customer stories and storyboarding as an integral part of her design process and is focused on making user-centered design more agile.

Thyra is an active member of the Usability Professionals' Association and is currently serving a second term on its board of directors as Secretary/Treasurer. She has a Ph.D. in experimental psychology from North Carolina State University, and is the author of 10 patents relating to user interface design.

Jacqueline Brodie

Doctoral candidate

Department of Information Systems and Computing

Brunel University, UK

Jacqueline is a doctoral candidate in the Department of Information Systems and Computing, Brunel University, UK. Jacqueline's research interests lie in methods for the design and implementation of mobile technology, as well as usability evaluation methods. Her publications cover both quantitative and qualitative research methods and she is currently on the Research and Strategy Committee of the British HCI group.

REFERENCES

Bailey, R. W. (2001). Heuristic evaluation vs User testing. UI design update newsletter - January 2001, http://www.humanfactors.com/downloads/jan01.asp.

Cockton, G. & Woolrych (2002). Sales must end: should discount methods be cleared off HCI's shelves? Interactions , September and October issue.

Molich, R. & Robin, J. (2003). Comparative Expert Reviews. In Proceedings of CHI'2003 Conference on Human Factors in Computing Systems Extended Abstracts, (pp.1060 – 1061, Washington, DC: ACM Press.

Molich, R., Thomsen, A. D., Karyukina, B., Schmid, L., Ede, M., van Oel, W., & Arcuri, M. (1999). Comparative evaluation of usability tests. Proceedings of CHI'99 Extended Abstracts , 83-84, Washington, DC: ACM Press.