Tutorials

This class will equip you to plan and carry out user studies in a culture outside your own. Heavy emphasis is placed on the practical steps involved in planning, since this is the most critical aspect of such studies. You will practice planning and see a video of user testing in Hong Kong.

PARTICIPANT KNOWLEDGE AND EXPERIENCE EXPECTED:

This class is aimed at people with experience doing user studies in their own country who want to learn how to conduct such studies in other countries and/or cultures.  This is intended to be a very practical session, intended primarily for practitioners, with links to academic work as appropriate. This class will NOT provide attendees with a set of "approved" facilities or contacts, although it will give them guidance on how to build a set of contacts for themselves.

GOALS FOR THE SESSION:

Participants in this tutorial will:

• Learn how to plan an international user study (including a field study, usability study, etc.) including:
• How to prepare your organization to do an international user study
• How to modify a study to fit cultural constraints
• How to find and prepare a location
• How to train the team and prepare yourself
• Identify what to expect on-site and when you return
• Practice planning an international user study

HOW THIS TUTORIAL WILL BE CONDUCTED

This tutorial will include a mix of lecture, discussion (both small group and large group), and hands-on exercises. We believe that learners learn best when engaged and participating, so at least half of the class will be discussion and exercises. A short video of a staged usability evaluation in Hong Kong will be shown for illustration, and will be followed by a discussion of the issues in simultaneous translation and facilitation presented in this film clip. The instructor will also include many case examples from her extensive experience doing cross-cultural studies.

TUTORIAL SCHEDULE WITH TIME ALLOCATION

DETAILED DESCRIPTION

8:30 – 10:00 am

(5 minutes) We will begin by reviewing the session structure, and asking several questions of the participants – including questions about what country they are from, their previous experience with user research (including international user research, if any) and international travel, industry sector and role in their organization, in order to target the remainder of the course to the actual participants' concerns and challenges to the extent possible. This sets the stage for a very interactive class.

(15 minutes) Participants will then learn about some of the reasons for doing international user studies. This works well as a short class discussion, since many people are beginning to explore

this type of study, and some have begun to do them. This further allows the instructor to

get to know the class, and vice versa. We have found that beginning in this way really increases interaction, which is one of our goals.

(40 minutes) Next, the instructor will introduce the critical elements of preparation. In our experience, it is preparation that makes the difference in the success or failure of an international study – especially the first time a person or organization does one. Therefore, the bulk of the material we cover is on preparation.

The first part of preparation is the organizational “selling” of an international user study. This is often the point where a user-interface person or manager has to convince others of the need to do such a study – therefore, the participants will learn about the elements of a strategy for doing this (which also nicely reinforces the reasons for doing an international user study in the first place), as well as who needs to be involved in the study.

(30 minutes) After covering this material in a lecture and discussion, attendees will break into small groups to come up with a strategy which they feel would work in their organization(s). If there are different approaches required (as is usually the case), this is a great way to highlight the need to tailor them to the organization. We will have a checklist for participants to help guide them in this process, which will be the basis of the exercise. We will have a discussion of the outcome of the exercise before a break.

10:30 – 12:00 noon

(30 minutes) After the break, we will discuss how to prepare the study itself. Often, user research must be modified in order to be carried out in another country/culture. Participants will learn about how to make decisions about where to go, and some of the key variables that affect the research design itself. For instance, the “normal procedures” for doing research vary widely in different parts of the world. It is important to understand if it is possible to tape and/or photograph sessions, for instance, as well as how much time to allow in between sessions. We will also discuss some of the key trade-offs in study design, as well as how to find a market research facility to help attendees with recruiting and facilities.

(15 minutes for both exercise and discussion) Participants will learn about tradeoffs in choosing a firm and then, attendees will break into small groups again and do an exercise where each group will make tradeoffs between two firms for a project in Paris. This will be followed by a class discussion.

(40 minutes) Next participants will learn about how to manage the mechanics of working with a firm in another country and handle more of the tradeoffs, such as what language to use for testing/research and how to keep track of all the details.

Once the location and firm has been decided, one must next prepare them for the study. This is the next topic we will handle. A key element will often be translation and interpretation. This is often the “weak link” in an international study, and we will cover these areas in detail. Participants will also learn about screening, facilitation and training of facilitators and translators, as well as other questions to ask to help attendees with preparation for the study.

(5 minutes) This section will wrap up with a general review of the material so far and will adjourn to lunch.

1:30 – 3:00 pm

(40 minutes) After lunch, participants will learn about how to prepare a team for an international study. We have found that usually, a group of people from the team will attend or help with the study. It is critically important that they be prepared to do this because so much hinges on the rapid integration of large amounts of often-confusing information. We will provide a framework for preparing the team for doing this, based on Nancy Adler's Cultural Synergy model, an approach that encourages cross-cultural communication.

(10 minutes) Our next exercise will be a “translation” exercise, where we will take phrases from slang (US, Canadian, UK English, as well as African, Asian and European examples) and have the class put them into Global English. Native speakers of English are often not aware of how much they use colloquial expressions, so this is, in part, a consciousness-raising experience for native speakers. For non-native speakers in the class, it is a validation and an opportunity to discuss their perceptions and experiences. The exercise inevitably evokes a lively discussion and a number of additional examples from participants.

(20 minutes) Once a locale and facility are chosen and prepared, and the team is ready, it is critical that the team leader be prepared, both at the lowest level (what phone plug do I need?) and at the highest (preparing for culture shock). Participants will learn about how to manage these preparations, and we will give them a “punch list” that we have found very useful in planning last minute details for a study.

(20 minutes for both individual portion of the exercise and class discussion) One of the key things for any leader to be aware of is his or her own expectations of the target locale. This is especially important when carrying out studies in new locales. In the "Expectations" exercise we will have participants choose a “candy” from a tray of Japanese, Indian, African and other “treats.” Each person writes down what they expect it will taste like individually, and then they taste it, and write down what it actually does taste like. We will then discuss the experience as a class. This is a fun exercise, which helps people recognize their own expectations while also helping maintain engagement in the materials.

3:30 – 5:00 pm

(30 minutes) At this point in the process, all is set – and the next step is to go and do the study. Therefore, we reconvene with a lecture and discussion about what to expect in-country, including many of the potential problems attendees may encounter and things they can do to cope with them. These are based on our own experiences on the road. We will facilitate a class discussion of other issues that may arise.

(10 minute) At this point, we will show a short videotape of a website evaluation we did in Hong Kong. This is a staged re-creation, showing a usability evaluation of 3 public sites, with the facilitator we used for the actual usability evaluations of a prototype redesign of a site, and a colleague of his as the evaluator. This tape illustrates a number of the points we make earlier – the issues of translation, the need to train the facilitator, the types of facilities one might expect, etc.

(10 minutes) After viewing the tape, we will have a class discussion of the issues it has raised.

(10 minutes) Next will be a discussion of coping on a personal level. Jet lag, stress, and culture shock are all potential risks that we will discuss, along with communication. To close out this section, we will discuss what to do if someone is sick or injured while doing an overseas study. This is a potentially terrifying experience (we know – it's happened to us!) and this is intended to help make it a little less difficult to cope with.

(15 minutes) The final section deals with what to expect coming home. Participants will learn about the wrap-up details and some of the things to expect in the organization. There will be a short individual exercise on “lessons learned” and initial thoughts on how to integrate or pull together the information from a study within each individual's organization.

(10 minutes) We will end with final questions, and a class discussion of the "lessons learned" by the participants.

(5 minutes) We will close with a review of the material of the day in a "fast-forward replay" followed by tutorial evaluation by the participants.

SPEAKER BIOS

Susan M. Dray, Ph.D.

President

Dray & Associates, Inc.

Since 1979, Susan has worked in the field of human factors to increase the quality and intuitiveness of user interface designs for users around the world. She consults internationally on interface design and usability. She combines expertise in contextual and ethnographic research and usability evaluation with a cross-cultural perspective, gained from consulting and teaching in 18 countries.

Susan is a leader in the Human Factors profession nationally and internationally. She has given over 100 talks at conferences and symposia in the U.S., Europe, and Australia, including plenary and keynote addresses at many international professional conferences. She is also a frequent presenter of tutorials at these conferences. In addition, she has published numerous papers and book chapters. Her most recent book chapter, written with David Siegel, is on planning and conducting international user studies. Together with David Siegel, she currently co-edits the Business Column of the Association for Computing Machinery's magazine interactions.

Susan was elected a Fellow of the Human Factors and Ergonomics Society and has chaired both the Organizational Design and Management Technical Group and the Computer Systems Technical Group of this same organization, as well as the Computers and Communications Scientific and Technical Committee of the International Ergonomics Association. Susan also served as North American editor of the prestigious international journal Behaviour and Information Technology.

Before starting her consulting firm, She was Director of Human Factors at IDS (now American Express Financial Advisors), where, in 1988, she developed one of the first corporate usability labs outside the computer industry. Previously, at Honeywell, she was involved in evaluating usability of consumer product hardware and software, as well as military technologies. Dr. Dray received her doctorate in Psychology from UCLA and is a Board Certified Human Factors Professional.

David A. Siegel, Ph.D.

Vice President

Dray & Associates, Inc.

David has worked as Vice President of Dray & Associates since 1993. He consults on usability, user interface design, and user-centered design processes. He specializes in conducting contextual user studies, as well as formal usability testing, expert evaluation of interface designs, and interface design consultation. He has experience carrying out studies both in the US and overseas.

He has published numerous articles and taught on a variety of user-centered design topics, including tutorials at professional conferences in the U.S., Europe, and Africa. He is the co-editor of the Business Column in ACM's magazine, interactions. He has recently co-authored (with Susan Dray) a chapter on planning international user studies, for Usability and Internationalization of Information Technology, edited by Nuray Ayken, and to be published by Lawrence Erlbaum Associates, Inc. this coming year.

David has helped product teams from a long list of clients to make pivotal improvements in their designs and product strategies based on user-centered research. Together with Susan, he conducted a series of iterative longitudinal field trials of successive prototypes of Microsoft's Tablet PC. These trials led to significant modification of the interactions design, form factor, and market positioning of the product. He played a leading role in designing the interface for Hewlett-Packard's first digital camera and for its desktop software, based on a process of iterative rapid prototyping with user testing at each stage. In awarding the resulting product "Best Buy" status, PC Magazine specifically praised it for its ease of use.