Tutorials

All usability professionals elicit information from users, through activities such as interviews, contextual inquiry, focus groups, usability tests, and surveys. While collecting information is easy, collecting the right information – and knowing that it is sufficient and unbiased – is very hard. This tutorial will help beginning and experienced practitioners to polish their elicitation skills with participatory exercises and examples.

PARTICIPANT KNOWLEDGE AND EXPERIENCE EXPECTED

This tutorial will not explain how to set up and run standard elicitation sessions such as focus groups, usability tests, contextual inquiry, interviews, or surveys. Participants should have at least basic knowledge of how these sessions are run. This tutorial will instead give participants skills they can use while running such elicitations. This tutorial is suitable for both junior and senior practitioners, because these are the skills that should be learned early, but practiced and polished throughout a career.

GOALS FOR THE SESSION:

This tutorial will teach participants specific skills that can be immediately applied to all situations where they elicit information from users. These skills include:

• Determining what information you need to collect. Ensuring you will learn everything you need to, and not wasting time collecting information you can’t use.
• Asking questions that will give you good data. Recognizing questions that are subtly or overtly leading. Avoiding ill-formed questions that will result in ambiguous or misleading data.
• Evaluating the quality of data that you get from other sources, by examining their elicitation protocols
• Handling the biasing effects of politics and basic sociology in focus groups and other multi-person elicitations.
• Intervening in usability tests without biasing the test results.

HOW THIS TUTORIAL WILL BE CONDUCTED

The tutorial is half lecture and half practical participatory exercises. Three speakers will take turns presenting their sections of the talk. This will include the points to be covered, along with anecdotal examples illustrating the points. All the anecdotes are from real-world practical experience. Each section will end with a practical exercise designed to reinforce the points in that section, and to provide participants with immediate practice putting their new knowledge to work.

Participants will be divided into groups of four or five to work on the exercises throughout the day. There are a variety of exercise types; some involve full-class participation (e.g., recognizing ill-formed questions) and others require working in groups (e.g., coming up with alternatives). Where possible the exercises allow participants to use examples from their own work (e.g., identifying decisions and goals, and determining what information is needed to support them).

The last section of the day consists of a series of video exercises. Participants will be shown short (staged) videos of elicitation sessions, and asked to identify errors in the protocol based on what they have learned. Participants will discuss how the protocols could be improved, then watch an “improved” video to contrast. (We have found that even quiet participants nervous about role-playing and other public exercises will participate fully in this exercise.)

There will be one final exercise that ties together everything covered in the day in a practical manner, to cement the new knowledge. Participants will be asked in their groups to design an elicitation session and the required questions to obtain particular information.

DETAILED DESCRIPTION OF TUTORIAL

Introduction

• Defining “elicitation”.
• Why it is so hard to elicit good data (how human nature, sociology, and bad planning gets in the way).
• Video: (3 minutes) Show a humorous worst-case scenario of bad elicitation.

Goals, Decisions and Information

• Knowing your goals and what decisions you need to make will determine what information you need to gather. (Discuss, lots of examples)
• Knowing the difference between decisions and information.
• The pitfalls of gathering information without understanding goals & decisions.
• Exercise 1. Understanding the difference between true decisions that need to be made, and information that needs to be gathered. (Full class)
• Exercise 2. Participants work in groups of four. Each person contributes an example of a real decision that needs to be made as part of his or her job, and the underlying goal. Then, the group determines types of information needed to make each decision. The class discusses a few examples.

Break

Data gathering

• Different ways to gather information (open questions, closed questions, metaphor questions, emotion card sort, forced rankings, etc.)
• When it is appropriate to use each kind of data gathering activity, and how to use it correctly. Recognizing and avoiding common misuses.
• Exercise 3. (Full class) Participants learn to recognize leading questions, and how to replace them with better elicitation that will provide unbiased data.
• Recognizing and replacing other badly-formed questions that can give you unusable data, such as two questions disguised as one, improperly worded questions, unqualified comparisons, over-demand of recall, partial examples, etc.
• Exercise 4. (Full class) Participants are presented with a badly-formed question, and discuss better ways to get the information.
• Exercise 5. Each group is given a different set of ill-formed questions. They work together to come up with alternative ways to get the desired information.
  

Lunch

Elicitation Sessions

• Understanding how and why normal human behavior can bias many kinds of elicitation sessions.
• Understanding the difference between opinion and behavior, and understanding what kinds of elicitation provide each kind of information.
• When and how you should elicit opinions, and when and how you should elicit actual behaviors
• Different kinds of elicitation sessions (e.g., focus groups, usability tests, contextual inquiry, interviews, surveys, etc), and what each kind is good for.
• Examples of inappropriate elicitations, and why they lead to bad information.
• Exercise 6. (Full class) Go back to the decision and information lists that were made up in exercise 2. As a class, select the correct type of elicitation that can best collect each piece of information.
• Common errors to avoid when planning an elicitation session.

Handling Common Problems

• Knowing when you can safely intervene in an elicitation session, how to do so with minimal impact, and what to do with the data afterwards.
• How to recover from lapses in protocol, and how to avoid them in the first place.
• Seeing the big picture – when to move off protocol.
• Avoiding pre-processing – not letting your preconceptions color your data analysis.
• Dealing with external issues (e.g., bugs, crashes, interfering observers, late-comers, etc.)
• Dealing with sociology problems in group-elicitation sessions (e.g., politics, dominant types, groupthink, run-on talkers, mumblers, problematic participants, etc.)
• Dealing with participants who confound or give the wrong data (i.e., unrepresentative users, liars, or participants with an agenda).

Break

Final Exercises

• Exercise 7. Participants are shown a series of short videos, each one containing a number of common errors of protocol. After each video, the class identifies the problems, and discusses alternative ways the desired information could have been collected. Then, an “improved” version of the same situation is shown.
• Exercise 8. The course presenters give participants an example of a decision that needs to be made, and a goal that the decision supports. Using knowledge from the course, participants work in their groups to plan an appropriate elicitation to get the information needed for that decision. The presenters move from table to table to help where needed, as participants go through the whole process of identifying what information is needed, and how to design the instrument to collect that information. Groups will then collect the information from each other.
  

SPEAKERS BIOS

Lynn Miller
Director of User Interface Development, Alias

Lynn Miller is the Director of User Interface Development at Alias, the world’s leading provider of 3D software for design, game creation and graphical special effects for film and television. She has been working as a usability professional for more than ten years, and has managed the Alias Usability Team for the last 5 years. She has taught courses in interface design, and has presented at UPA.

John Schrag
Interaction Designer, Alias

John Schrag is an Interaction Designer at Alias. During his 14 years there, he has worked primarily on the UI design and architecture of new products for 3D animation and visualization. Over his career John has taught courses (both in-house and at ACM SIGGRAPH) in user interface design, usability practice, and designing 3D graphical interfaces.


Desirée Sy
Interaction Designer, Alias

Desirée Sy is an Interaction Designer at Alias. She has been working as a usability professional for more than 10 years. She created and taught a well-rated 3-day course on how to conduct low-cost usability studies for Rockley Consulting Services, and has presented at ACM SIGDOC, IPCC, and UPA.